[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:03:54.357049 5099 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.250.254:38841
I20240503 14:03:54.358597 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:03:54.359809 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:54.380259 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:54.380316 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:54.382032 5099 server_base.cc:1034] running on GCE node
W20240503 14:03:54.380918 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:55.439175 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:55.439453 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:55.439661 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745035439636 us; error 0 us; skew 500 ppm
I20240503 14:03:55.440570 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:55.451582 5099 webserver.cc:460] Webserver started at http://127.4.250.254:45997/ using document root <none> and password file <none>
I20240503 14:03:55.452960 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:55.453284 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:55.453877 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:55.462674 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/master-0-root/instance:
uuid: "e7509361dc6d4b9988670406b4ecd2f4"
format_stamp: "Formatted at 2024-05-03 14:03:55 on dist-test-slave-8wrt"
I20240503 14:03:55.471560 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20240503 14:03:55.477241 5114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:55.478591 5099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240503 14:03:55.479015 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/master-0-root
uuid: "e7509361dc6d4b9988670406b4ecd2f4"
format_stamp: "Formatted at 2024-05-03 14:03:55 on dist-test-slave-8wrt"
I20240503 14:03:55.479403 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:55.542351 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:55.543951 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:03:55.544462 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:55.626137 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.254:38841
I20240503 14:03:55.626242 5165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.254:38841 every 8 connection(s)
I20240503 14:03:55.634636 5166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:55.661329 5166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: Bootstrap starting.
I20240503 14:03:55.667634 5166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:55.669685 5166 log.cc:824] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: Log is configured to *not* fsync() on all Append() calls
I20240503 14:03:55.675184 5166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: No bootstrap required, opened a new log
I20240503 14:03:55.700466 5166 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER }
I20240503 14:03:55.701103 5166 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:55.701318 5166 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7509361dc6d4b9988670406b4ecd2f4, State: Initialized, Role: FOLLOWER
I20240503 14:03:55.702718 5166 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER }
I20240503 14:03:55.703256 5166 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:03:55.703521 5166 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:03:55.703863 5166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:55.708321 5166 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER }
I20240503 14:03:55.709209 5166 leader_election.cc:302] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7509361dc6d4b9988670406b4ecd2f4; no voters:
I20240503 14:03:55.711498 5166 leader_election.cc:288] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:03:55.711836 5169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:55.715381 5169 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 1 LEADER]: Becoming Leader. State: Replica: e7509361dc6d4b9988670406b4ecd2f4, State: Running, Role: LEADER
I20240503 14:03:55.716476 5169 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER }
I20240503 14:03:55.718093 5166 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:03:55.730311 5170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7509361dc6d4b9988670406b4ecd2f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER } }
I20240503 14:03:55.730311 5171 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7509361dc6d4b9988670406b4ecd2f4. Latest consensus state: current_term: 1 leader_uuid: "e7509361dc6d4b9988670406b4ecd2f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7509361dc6d4b9988670406b4ecd2f4" member_type: VOTER } }
I20240503 14:03:55.731243 5170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [sys.catalog]: This master's current role is: LEADER
I20240503 14:03:55.731264 5171 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [sys.catalog]: This master's current role is: LEADER
I20240503 14:03:55.735150 5179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:03:55.749936 5179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:03:55.751873 5099 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:03:55.769503 5179 catalog_manager.cc:1347] Generated new cluster ID: 2279d09e253c4339a0e8903de03b99be
I20240503 14:03:55.769886 5179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:03:55.793025 5179 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:03:55.795357 5179 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:03:55.819944 5179 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: Generated new TSK 0
I20240503 14:03:55.821252 5179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:03:55.884481 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:55.894824 5188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:55.896420 5189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:55.901780 5191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:55.902323 5099 server_base.cc:1034] running on GCE node
I20240503 14:03:55.903262 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:55.903568 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:55.903775 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745035903758 us; error 0 us; skew 500 ppm
I20240503 14:03:55.904702 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:55.909760 5099 webserver.cc:460] Webserver started at http://127.4.250.193:40137/ using document root <none> and password file <none>
I20240503 14:03:55.910562 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:55.910845 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:55.911197 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:55.912887 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-0-root/instance:
uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
format_stamp: "Formatted at 2024-05-03 14:03:55 on dist-test-slave-8wrt"
I20240503 14:03:55.920889 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20240503 14:03:55.926822 5196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:55.927721 5099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:03:55.928068 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
format_stamp: "Formatted at 2024-05-03 14:03:55 on dist-test-slave-8wrt"
I20240503 14:03:55.928444 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:55.952404 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:55.954231 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:55.956990 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:55.961658 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:55.961901 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:55.962201 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:55.962368 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.077358 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.193:43935
I20240503 14:03:56.077442 5258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.193:43935 every 8 connection(s)
I20240503 14:03:56.086813 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:56.102890 5263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:56.103071 5264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:56.114886 5266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:56.117368 5099 server_base.cc:1034] running on GCE node
I20240503 14:03:56.118413 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:56.118712 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:56.118916 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745036118898 us; error 0 us; skew 500 ppm
I20240503 14:03:56.119746 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:56.123318 5259 heartbeater.cc:347] Connected to a master server at 127.4.250.254:38841
I20240503 14:03:56.123873 5259 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:56.125798 5259 heartbeater.cc:511] Master 127.4.250.254:38841 requested a full tablet report, sending...
I20240503 14:03:56.129043 5099 webserver.cc:460] Webserver started at http://127.4.250.194:36457/ using document root <none> and password file <none>
I20240503 14:03:56.129652 5131 ts_manager.cc:194] Registered new tserver with Master: 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935)
I20240503 14:03:56.129890 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:56.130193 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:56.130584 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:56.132037 5131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43174
I20240503 14:03:56.132200 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-1-root/instance:
uuid: "d3ea46436f134d77b24e334796af1c56"
format_stamp: "Formatted at 2024-05-03 14:03:56 on dist-test-slave-8wrt"
I20240503 14:03:56.139900 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240503 14:03:56.145583 5271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.146670 5099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20240503 14:03:56.147100 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
uuid: "d3ea46436f134d77b24e334796af1c56"
format_stamp: "Formatted at 2024-05-03 14:03:56 on dist-test-slave-8wrt"
I20240503 14:03:56.147501 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:56.186870 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:56.188661 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:56.190871 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:56.194636 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:56.194929 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.195217 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:56.195433 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.305773 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.194:40355
I20240503 14:03:56.305899 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.194:40355 every 8 connection(s)
I20240503 14:03:56.313448 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:03:56.320914 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:03:56.323035 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:56.331034 5334 heartbeater.cc:347] Connected to a master server at 127.4.250.254:38841
I20240503 14:03:56.331506 5334 heartbeater.cc:464] Registering TS with master...
W20240503 14:03:56.331960 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:03:56.332417 5099 server_base.cc:1034] running on GCE node
I20240503 14:03:56.332495 5334 heartbeater.cc:511] Master 127.4.250.254:38841 requested a full tablet report, sending...
I20240503 14:03:56.333508 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:03:56.333787 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:03:56.333951 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745036333934 us; error 0 us; skew 500 ppm
I20240503 14:03:56.334707 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:03:56.335197 5131 ts_manager.cc:194] Registered new tserver with Master: d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:56.336584 5131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43184
I20240503 14:03:56.339293 5099 webserver.cc:460] Webserver started at http://127.4.250.195:34359/ using document root <none> and password file <none>
I20240503 14:03:56.339982 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:03:56.340265 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:03:56.340611 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:03:56.342084 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-2-root/instance:
uuid: "d25004c4b8f84302b323f34a58117752"
format_stamp: "Formatted at 2024-05-03 14:03:56 on dist-test-slave-8wrt"
I20240503 14:03:56.348076 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:03:56.353695 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.354568 5099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.001s
I20240503 14:03:56.354872 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
uuid: "d25004c4b8f84302b323f34a58117752"
format_stamp: "Formatted at 2024-05-03 14:03:56 on dist-test-slave-8wrt"
I20240503 14:03:56.355180 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714745034190430-5099-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:03:56.369421 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:03:56.370719 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:03:56.373418 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:03:56.376614 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:03:56.376855 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.377103 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:03:56.377270 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:03:56.484817 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.195:42405
I20240503 14:03:56.485564 5407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.195:42405 every 8 connection(s)
I20240503 14:03:56.503700 5408 heartbeater.cc:347] Connected to a master server at 127.4.250.254:38841
I20240503 14:03:56.504069 5408 heartbeater.cc:464] Registering TS with master...
I20240503 14:03:56.504902 5408 heartbeater.cc:511] Master 127.4.250.254:38841 requested a full tablet report, sending...
I20240503 14:03:56.506801 5131 ts_manager.cc:194] Registered new tserver with Master: d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:56.507246 5099 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014873456s
I20240503 14:03:56.508684 5131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43188
I20240503 14:03:56.511605 5099 test_util.cc:268] Using random seed: 818420048
I20240503 14:03:56.569677 5131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43194:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:03:56.574030 5131 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:03:56.693861 5220 tablet_service.cc:1467] Processing CreateTablet for tablet a76eec11ab4f41a8a14915ab78fbdc2f (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 14:03:56.698789 5220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76eec11ab4f41a8a14915ab78fbdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.701252 5222 tablet_service.cc:1467] Processing CreateTablet for tablet 11308338f3fe4e4a9aef449bbd587c34 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 14:03:56.703080 5218 tablet_service.cc:1467] Processing CreateTablet for tablet 03e7042a091e45c8b75cbbc88c950e67 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 14:03:56.702059 5219 tablet_service.cc:1467] Processing CreateTablet for tablet e09a2a31688a4a699253667f00b716b2 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 14:03:56.700474 5223 tablet_service.cc:1467] Processing CreateTablet for tablet 1c08089db36f41f19299ca62b01986ec (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 14:03:56.702026 5221 tablet_service.cc:1467] Processing CreateTablet for tablet 4454a646fb5a4246abe5215f17d099bb (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 14:03:56.702168 5224 tablet_service.cc:1467] Processing CreateTablet for tablet 22097a93651744deb19d0f1a634b5817 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 14:03:56.709642 5217 tablet_service.cc:1467] Processing CreateTablet for tablet c570d2dbff30430290e8f782ad2ea0d1 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 14:03:56.711971 5222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11308338f3fe4e4a9aef449bbd587c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.715691 5216 tablet_service.cc:1467] Processing CreateTablet for tablet aacb2e13659140d5a7b52329f9c578db (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 14:03:56.717490 5223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c08089db36f41f19299ca62b01986ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.719269 5221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454a646fb5a4246abe5215f17d099bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.721374 5219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a2a31688a4a699253667f00b716b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.724138 5224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22097a93651744deb19d0f1a634b5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.726352 5216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacb2e13659140d5a7b52329f9c578db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.727033 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570d2dbff30430290e8f782ad2ea0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.728425 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e7042a091e45c8b75cbbc88c950e67. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:03:56.735810 5178 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:03:56.751905 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 22097a93651744deb19d0f1a634b5817 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 14:03:56.762984 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 11308338f3fe4e4a9aef449bbd587c34 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 14:03:56.766986 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 1c08089db36f41f19299ca62b01986ec (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 14:03:56.771714 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22097a93651744deb19d0f1a634b5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.773479 5297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11308338f3fe4e4a9aef449bbd587c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.774580 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c08089db36f41f19299ca62b01986ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.786113 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 4454a646fb5a4246abe5215f17d099bb (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 14:03:56.786186 5295 tablet_service.cc:1467] Processing CreateTablet for tablet a76eec11ab4f41a8a14915ab78fbdc2f (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 14:03:56.800256 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 03e7042a091e45c8b75cbbc88c950e67 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 14:03:56.798547 5292 tablet_service.cc:1467] Processing CreateTablet for tablet c570d2dbff30430290e8f782ad2ea0d1 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 14:03:56.807140 5291 tablet_service.cc:1467] Processing CreateTablet for tablet aacb2e13659140d5a7b52329f9c578db (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 14:03:56.810350 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76eec11ab4f41a8a14915ab78fbdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.812501 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e7042a091e45c8b75cbbc88c950e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.810693 5294 tablet_service.cc:1467] Processing CreateTablet for tablet e09a2a31688a4a699253667f00b716b2 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 14:03:56.815099 5296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454a646fb5a4246abe5215f17d099bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.818845 5291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacb2e13659140d5a7b52329f9c578db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.823742 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a2a31688a4a699253667f00b716b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.838591 5371 tablet_service.cc:1467] Processing CreateTablet for tablet 11308338f3fe4e4a9aef449bbd587c34 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240503 14:03:56.840617 5371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11308338f3fe4e4a9aef449bbd587c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.835062 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570d2dbff30430290e8f782ad2ea0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.836892 5370 tablet_service.cc:1467] Processing CreateTablet for tablet 4454a646fb5a4246abe5215f17d099bb (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240503 14:03:56.843180 5372 tablet_service.cc:1467] Processing CreateTablet for tablet 1c08089db36f41f19299ca62b01986ec (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240503 14:03:56.837677 5373 tablet_service.cc:1467] Processing CreateTablet for tablet 22097a93651744deb19d0f1a634b5817 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240503 14:03:56.855522 5372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c08089db36f41f19299ca62b01986ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.855564 5369 tablet_service.cc:1467] Processing CreateTablet for tablet a76eec11ab4f41a8a14915ab78fbdc2f (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240503 14:03:56.866891 5368 tablet_service.cc:1467] Processing CreateTablet for tablet e09a2a31688a4a699253667f00b716b2 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240503 14:03:56.869484 5373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22097a93651744deb19d0f1a634b5817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.876412 5367 tablet_service.cc:1467] Processing CreateTablet for tablet 03e7042a091e45c8b75cbbc88c950e67 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240503 14:03:56.882833 5368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a2a31688a4a699253667f00b716b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.890463 5426 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:56.901252 5370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4454a646fb5a4246abe5215f17d099bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.899734 5366 tablet_service.cc:1467] Processing CreateTablet for tablet c570d2dbff30430290e8f782ad2ea0d1 (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240503 14:03:56.914848 5365 tablet_service.cc:1467] Processing CreateTablet for tablet aacb2e13659140d5a7b52329f9c578db (DEFAULT_TABLE table=test-workload [id=ce16a9a994224429aeed4d11a0c1e866]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240503 14:03:56.922771 5369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76eec11ab4f41a8a14915ab78fbdc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.940543 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c570d2dbff30430290e8f782ad2ea0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.943441 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacb2e13659140d5a7b52329f9c578db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:56.947412 5367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e7042a091e45c8b75cbbc88c950e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:03:57.023373 5427 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.064011 5427 tablet_bootstrap.cc:654] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.064038 5428 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.078966 5426 tablet_bootstrap.cc:654] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.098115 5427 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.098664 5427 ts_tablet_manager.cc:1367] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.077s user 0.019s sys 0.000s
I20240503 14:03:57.106559 5426 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.106992 5426 ts_tablet_manager.cc:1367] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.218s user 0.004s sys 0.015s
I20240503 14:03:57.111342 5428 tablet_bootstrap.cc:654] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.112566 5427 raft_consensus.cc:348] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.114413 5427 raft_consensus.cc:374] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.115013 5427 raft_consensus.cc:729] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.114364 5426 raft_consensus.cc:348] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.116745 5426 raft_consensus.cc:374] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.117070 5426 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.117866 5426 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.115775 5427 consensus_queue.cc:260] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.124187 5428 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.124125 5259 heartbeater.cc:503] Master 127.4.250.254:38841 was elected leader, sending a full tablet report...
I20240503 14:03:57.124558 5426 ts_tablet_manager.cc:1398] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.017s user 0.001s sys 0.010s
I20240503 14:03:57.124971 5428 ts_tablet_manager.cc:1367] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.012s
I20240503 14:03:57.126214 5426 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.129992 5427 ts_tablet_manager.cc:1398] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.031s user 0.010s sys 0.000s
I20240503 14:03:57.130631 5334 heartbeater.cc:503] Master 127.4.250.254:38841 was elected leader, sending a full tablet report...
I20240503 14:03:57.131088 5427 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.141127 5426 tablet_bootstrap.cc:654] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.143860 5427 tablet_bootstrap.cc:654] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.140982 5428 raft_consensus.cc:348] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.147330 5428 raft_consensus.cc:374] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.147650 5428 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.148420 5428 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.154696 5428 ts_tablet_manager.cc:1398] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20240503 14:03:57.154834 5408 heartbeater.cc:503] Master 127.4.250.254:38841 was elected leader, sending a full tablet report...
I20240503 14:03:57.155683 5428 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.163561 5428 tablet_bootstrap.cc:654] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.170531 5426 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.170948 5426 ts_tablet_manager.cc:1367] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.007s
I20240503 14:03:57.173848 5427 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.176488 5427 ts_tablet_manager.cc:1367] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.005s
I20240503 14:03:57.180423 5427 raft_consensus.cc:348] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.181310 5427 raft_consensus.cc:374] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.181623 5427 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.182459 5427 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.173316 5426 raft_consensus.cc:348] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.184835 5426 raft_consensus.cc:374] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.185232 5426 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.185977 5426 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.188311 5426 ts_tablet_manager.cc:1398] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.017s user 0.007s sys 0.003s
I20240503 14:03:57.189324 5426 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.195688 5426 tablet_bootstrap.cc:654] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.196640 5427 ts_tablet_manager.cc:1398] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20240503 14:03:57.197889 5427 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.204314 5427 tablet_bootstrap.cc:654] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.208747 5428 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.209270 5428 ts_tablet_manager.cc:1367] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.013s
I20240503 14:03:57.211966 5428 raft_consensus.cc:348] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.212869 5428 raft_consensus.cc:374] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.213187 5428 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.214359 5427 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.213486 5426 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.214860 5427 ts_tablet_manager.cc:1367] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20240503 14:03:57.215446 5426 ts_tablet_manager.cc:1367] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20240503 14:03:57.215240 5428 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.218225 5428 ts_tablet_manager.cc:1398] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20240503 14:03:57.219723 5428 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.219254 5427 raft_consensus.cc:348] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.220116 5427 raft_consensus.cc:374] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.220422 5427 raft_consensus.cc:729] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.221110 5427 consensus_queue.cc:260] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.223671 5427 ts_tablet_manager.cc:1398] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:03:57.224658 5427 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.231488 5427 tablet_bootstrap.cc:654] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.233408 5428 tablet_bootstrap.cc:654] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.242588 5426 raft_consensus.cc:348] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.243415 5426 raft_consensus.cc:374] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.243728 5426 raft_consensus.cc:729] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
W20240503 14:03:57.244058 5409 tablet.cc:2369] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:03:57.244485 5426 consensus_queue.cc:260] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.249173 5426 ts_tablet_manager.cc:1398] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20240503 14:03:57.250180 5426 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.256426 5426 tablet_bootstrap.cc:654] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.261847 5427 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.262333 5427 ts_tablet_manager.cc:1367] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.021s
I20240503 14:03:57.265257 5428 tablet_bootstrap.cc:492] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.265758 5428 ts_tablet_manager.cc:1367] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.009s
I20240503 14:03:57.265367 5427 raft_consensus.cc:348] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.266149 5427 raft_consensus.cc:374] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.266443 5427 raft_consensus.cc:729] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.267158 5427 consensus_queue.cc:260] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.267887 5428 raft_consensus.cc:348] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.268595 5428 raft_consensus.cc:374] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.268860 5428 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.269450 5427 ts_tablet_manager.cc:1398] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.007s user 0.001s sys 0.008s
I20240503 14:03:57.269486 5428 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.270928 5427 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.273304 5428 ts_tablet_manager.cc:1398] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240503 14:03:57.274211 5428 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.277208 5427 tablet_bootstrap.cc:654] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.280717 5428 tablet_bootstrap.cc:654] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.290824 5426 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.291329 5426 ts_tablet_manager.cc:1367] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.004s
I20240503 14:03:57.295158 5426 raft_consensus.cc:348] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.296612 5426 raft_consensus.cc:374] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.297417 5426 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.298382 5426 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.299526 5427 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.300211 5427 ts_tablet_manager.cc:1367] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.000s
I20240503 14:03:57.301232 5436 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.301767 5436 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.303552 5427 raft_consensus.cc:348] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.304358 5427 raft_consensus.cc:374] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.304682 5427 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.321180 5432 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.325588 5426 ts_tablet_manager.cc:1398] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.034s user 0.008s sys 0.002s
I20240503 14:03:57.322430 5436 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.323655 5432 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.331900 5426 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.324872 5427 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.335641 5432 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.341414 5427 ts_tablet_manager.cc:1398] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.041s user 0.012s sys 0.000s
I20240503 14:03:57.342412 5427 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.352856 5427 tablet_bootstrap.cc:654] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.356374 5436 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.357658 5436 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.360256 5436 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.387785 5428 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.388466 5428 ts_tablet_manager.cc:1367] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.114s user 0.013s sys 0.005s
I20240503 14:03:57.389145 5426 tablet_bootstrap.cc:654] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.394227 5428 raft_consensus.cc:348] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.395435 5428 raft_consensus.cc:374] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.398339 5428 raft_consensus.cc:729] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.399673 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.399780 5428 consensus_queue.cc:260] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.403149 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.406410 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.409046 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.409864 5428 ts_tablet_manager.cc:1398] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.021s user 0.005s sys 0.004s
I20240503 14:03:57.410166 5233 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 0.
I20240503 14:03:57.408749 5234 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 0.
I20240503 14:03:57.412531 5428 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.415382 5348 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:03:57.416286 5436 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.416649 5436 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.416976 5436 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.417305 5427 tablet_bootstrap.cc:492] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
W20240503 14:03:57.414535 5349 leader_election.cc:341] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.418401 5427 ts_tablet_manager.cc:1367] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.004s
W20240503 14:03:57.418828 5349 leader_election.cc:341] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.423075 5348 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters: d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.428105 5436 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.433982 5454 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.435173 5433 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.439898 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.448069 5428 tablet_bootstrap.cc:654] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.454500 5234 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.456224 5436 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.445699 5454 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.455940 5433 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.460592 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:57.453366 5456 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.459249 5454 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:03:57.463006 5349 leader_election.cc:341] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.465438 5427 raft_consensus.cc:348] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.467397 5427 raft_consensus.cc:374] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.468830 5433 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.469751 5427 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.472566 5427 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.465193 5456 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.482421 5436 raft_consensus.cc:482] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.482933 5436 raft_consensus.cc:504] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.485234 5436 leader_election.cc:288] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.490343 5456 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.495388 5426 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.501893 5427 ts_tablet_manager.cc:1398] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.083s user 0.008s sys 0.002s
I20240503 14:03:57.500185 5426 ts_tablet_manager.cc:1367] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.168s user 0.028s sys 0.017s
I20240503 14:03:57.513962 5427 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.517030 5426 raft_consensus.cc:348] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.518608 5426 raft_consensus.cc:374] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.518908 5426 raft_consensus.cc:729] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.522060 5426 consensus_queue.cc:260] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.526124 5426 ts_tablet_manager.cc:1398] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240503 14:03:57.527134 5426 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.530045 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.530573 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.531316 5233 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
I20240503 14:03:57.532841 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.534737 5234 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 1.
I20240503 14:03:57.538271 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.538419 5348 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters: d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.539563 5436 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.536543 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.542398 5427 tablet_bootstrap.cc:654] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
W20240503 14:03:57.542699 5349 leader_election.cc:341] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.536374 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
W20240503 14:03:57.544565 5348 leader_election.cc:341] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.545063 5348 leader_election.cc:302] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.548312 5456 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.549378 5456 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.552234 5456 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.549005 5436 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:57.555191 5428 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.556385 5428 ts_tablet_manager.cc:1367] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.144s user 0.036s sys 0.021s
I20240503 14:03:57.557724 5426 tablet_bootstrap.cc:654] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.558650 5274 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.559130 5436 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.561492 5456 raft_consensus.cc:2781] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:03:57.561944 5274 leader_election.cc:341] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.564116 5456 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.559288 5461 raft_consensus.cc:2726] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:03:57.565891 5200 leader_election.cc:341] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.564885 5456 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.559804 5428 raft_consensus.cc:348] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.567116 5428 raft_consensus.cc:374] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.568341 5428 raft_consensus.cc:729] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
W20240503 14:03:57.572788 5200 leader_election.cc:341] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.572252 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.573724 5200 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.573958 5232 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
I20240503 14:03:57.574661 5428 consensus_queue.cc:260] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.575304 5432 raft_consensus.cc:2726] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.572244 5454 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.576715 5461 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.577311 5274 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.577962 5461 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.579768 5433 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.580436 5433 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.580757 5433 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.581795 5428 ts_tablet_manager.cc:1398] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20240503 14:03:57.583334 5456 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.585613 5456 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.586557 5428 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.587934 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.588706 5232 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.592842 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.595000 5461 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.601089 5454 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.600704 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.595981 5433 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.602646 5308 raft_consensus.cc:2379] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d25004c4b8f84302b323f34a58117752 in current term 1: Already voted for candidate d3ea46436f134d77b24e334796af1c56 in this term.
I20240503 14:03:57.603195 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.604616 5234 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.603313 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:57.608424 5349 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.609630 5427 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.609800 5232 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.609992 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.610146 5427 ts_tablet_manager.cc:1367] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.107s user 0.007s sys 0.010s
I20240503 14:03:57.610689 5383 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
W20240503 14:03:57.611696 5349 leader_election.cc:341] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.612612 5274 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.613341 5427 raft_consensus.cc:348] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.613853 5456 raft_consensus.cc:2781] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.614197 5427 raft_consensus.cc:374] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.614562 5427 raft_consensus.cc:729] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.615078 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.615758 5383 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
I20240503 14:03:57.615401 5427 consensus_queue.cc:260] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.616132 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.615056 5234 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 1.
I20240503 14:03:57.617823 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:57.618346 5382 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.623803 5428 tablet_bootstrap.cc:654] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.624759 5433 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.626978 5275 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:03:57.628407 5433 raft_consensus.cc:2781] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:03:57.628584 5275 leader_election.cc:341] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.628863 5433 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.626549 5348 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters: d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.630355 5454 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.630821 5454 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:57.631594 5454 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.633500 5382 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.629603 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.634279 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:57.635396 5234 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.635952 5462 raft_consensus.cc:2726] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.629266 5433 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:03:57.638890 5275 leader_election.cc:341] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.638893 5456 raft_consensus.cc:686] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:57.652777 5456 consensus_queue.cc:237] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.661784 5234 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.662736 5428 tablet_bootstrap.cc:492] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.663280 5274 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters: d25004c4b8f84302b323f34a58117752
I20240503 14:03:57.663266 5428 ts_tablet_manager.cc:1367] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.021s
I20240503 14:03:57.667773 5466 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.673127 5466 raft_consensus.cc:686] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:57.674180 5466 consensus_queue.cc:237] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.681388 5428 raft_consensus.cc:348] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.683159 5428 raft_consensus.cc:374] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.684338 5428 raft_consensus.cc:729] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.685346 5428 consensus_queue.cc:260] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.682340 5131 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 0 to 1, leader changed from <none> to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:03:57.687625 5427 ts_tablet_manager.cc:1398] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.077s user 0.018s sys 0.010s
I20240503 14:03:57.687045 5426 tablet_bootstrap.cc:492] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.688033 5428 ts_tablet_manager.cc:1398] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20240503 14:03:57.690014 5426 ts_tablet_manager.cc:1367] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.163s user 0.022s sys 0.009s
I20240503 14:03:57.690115 5427 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.690655 5428 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.687695 5128 catalog_manager.cc:5484] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:57.695781 5426 raft_consensus.cc:348] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.697199 5426 raft_consensus.cc:374] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.697552 5426 raft_consensus.cc:729] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.699223 5426 consensus_queue.cc:260] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.705276 5426 ts_tablet_manager.cc:1398] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240503 14:03:57.707548 5426 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.708606 5433 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.710841 5427 tablet_bootstrap.cc:654] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.713232 5466 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.713600 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.713996 5433 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.715169 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:57.718381 5234 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.718976 5383 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.723407 5428 tablet_bootstrap.cc:654] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.722446 5466 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.733937 5234 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.735599 5383 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.735941 5274 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.736795 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.737975 5433 raft_consensus.cc:2781] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.736795 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.738498 5433 raft_consensus.cc:686] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
W20240503 14:03:57.738616 5178 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20240503 14:03:57.738867 5275 leader_election.cc:341] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:03:57.739674 5274 leader_election.cc:341] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.739385 5433 consensus_queue.cc:237] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.740165 5274 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:03:57.741065 5456 raft_consensus.cc:2726] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.740653 5428 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.742290 5428 ts_tablet_manager.cc:1367] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.006s
I20240503 14:03:57.737418 5466 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.746191 5426 tablet_bootstrap.cc:654] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.751875 5428 raft_consensus.cc:348] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.757949 5428 raft_consensus.cc:374] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.758431 5428 raft_consensus.cc:729] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.760298 5428 consensus_queue.cc:260] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.767093 5427 tablet_bootstrap.cc:492] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
I20240503 14:03:57.767851 5427 ts_tablet_manager.cc:1367] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.008s
I20240503 14:03:57.772061 5428 ts_tablet_manager.cc:1398] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20240503 14:03:57.773262 5428 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.774402 5426 tablet_bootstrap.cc:492] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.774950 5426 ts_tablet_manager.cc:1367] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.000s
I20240503 14:03:57.778623 5426 raft_consensus.cc:348] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.779510 5426 raft_consensus.cc:374] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.779858 5426 raft_consensus.cc:729] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.779858 5427 raft_consensus.cc:348] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.780795 5427 raft_consensus.cc:374] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.781119 5427 raft_consensus.cc:729] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.780968 5426 consensus_queue.cc:260] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.784474 5427 consensus_queue.cc:260] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.785835 5428 tablet_bootstrap.cc:654] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.787575 5427 ts_tablet_manager.cc:1398] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240503 14:03:57.790145 5426 ts_tablet_manager.cc:1398] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.015s user 0.007s sys 0.002s
I20240503 14:03:57.791252 5426 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.798187 5426 tablet_bootstrap.cc:654] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.799392 5433 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.800024 5433 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.800675 5427 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: Bootstrap starting.
I20240503 14:03:57.803491 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.804616 5232 raft_consensus.cc:2379] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3ea46436f134d77b24e334796af1c56 in current term 1: Already voted for candidate d25004c4b8f84302b323f34a58117752 in this term.
I20240503 14:03:57.808032 5427 tablet_bootstrap.cc:654] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.809130 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.811264 5275 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:03:57.812325 5466 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.812366 5433 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.794754 5131 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 0 to 1, leader changed from <none> to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:03:57.817281 5456 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.817854 5456 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.820933 5466 raft_consensus.cc:2726] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.822430 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.824002 5427 tablet_bootstrap.cc:492] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: No bootstrap required, opened a new log
W20240503 14:03:57.824268 5274 leader_election.cc:341] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.824559 5427 ts_tablet_manager.cc:1367] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.005s
I20240503 14:03:57.825363 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.819923 5456 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
W20240503 14:03:57.828693 5275 leader_election.cc:341] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:03:57.829388 5275 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:03:57.830350 5456 raft_consensus.cc:2726] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.831396 5427 raft_consensus.cc:348] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.832273 5427 raft_consensus.cc:374] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.832625 5427 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.833250 5426 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.833292 5427 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.835628 5427 ts_tablet_manager.cc:1398] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240503 14:03:57.833845 5426 ts_tablet_manager.cc:1367] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.009s
I20240503 14:03:57.840392 5129 catalog_manager.cc:5484] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:57.842867 5428 tablet_bootstrap.cc:492] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.842372 5129 catalog_manager.cc:5484] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:57.843413 5428 ts_tablet_manager.cc:1367] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.070s user 0.031s sys 0.011s
I20240503 14:03:57.844038 5426 raft_consensus.cc:348] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.844777 5426 raft_consensus.cc:374] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.845070 5426 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.845721 5426 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.847345 5426 ts_tablet_manager.cc:1398] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:03:57.848659 5426 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: Bootstrap starting.
I20240503 14:03:57.852018 5428 raft_consensus.cc:348] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.852946 5428 raft_consensus.cc:374] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.853310 5428 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.854055 5428 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.856396 5428 ts_tablet_manager.cc:1398] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20240503 14:03:57.859097 5426 tablet_bootstrap.cc:654] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.872371 5428 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: Bootstrap starting.
I20240503 14:03:57.875411 5426 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: No bootstrap required, opened a new log
I20240503 14:03:57.876021 5426 ts_tablet_manager.cc:1367] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.010s
I20240503 14:03:57.876744 5462 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.877519 5428 tablet_bootstrap.cc:654] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: Neither blocks nor log segments found. Creating new log.
I20240503 14:03:57.877300 5462 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.877789 5456 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.878275 5456 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.879113 5426 raft_consensus.cc:348] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.879990 5426 raft_consensus.cc:374] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.880334 5426 raft_consensus.cc:729] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.881057 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.881069 5426 consensus_queue.cc:260] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.882001 5232 raft_consensus.cc:2379] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d25004c4b8f84302b323f34a58117752 in current term 1: Already voted for candidate d3ea46436f134d77b24e334796af1c56 in this term.
I20240503 14:03:57.882402 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.883162 5383 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
I20240503 14:03:57.883963 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.884642 5232 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 0.
I20240503 14:03:57.885191 5275 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.886170 5433 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.886662 5456 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.887004 5426 ts_tablet_manager.cc:1398] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240503 14:03:57.886574 5433 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.887327 5436 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.887671 5433 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.888206 5462 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.887821 5436 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.886561 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.891160 5436 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.893126 5349 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:03:57.894637 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.895363 5309 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 0.
I20240503 14:03:57.897472 5433 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.898011 5349 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.898878 5462 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.900504 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.901157 5234 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.904368 5436 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.906030 5436 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.906494 5436 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.909117 5433 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.909796 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:57.910487 5383 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.912873 5432 raft_consensus.cc:482] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.913645 5428 tablet_bootstrap.cc:492] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: No bootstrap required, opened a new log
I20240503 14:03:57.913362 5432 raft_consensus.cc:504] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.914100 5428 ts_tablet_manager.cc:1367] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.013s
I20240503 14:03:57.915382 5432 leader_election.cc:288] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.905059 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.916602 5232 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 0.
I20240503 14:03:57.916934 5428 raft_consensus.cc:348] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.917728 5234 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.918164 5428 raft_consensus.cc:374] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:03:57.918464 5428 raft_consensus.cc:729] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Initialized, Role: FOLLOWER
I20240503 14:03:57.918759 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.919052 5428 consensus_queue.cc:260] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.919481 5382 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 0.
I20240503 14:03:57.921108 5428 ts_tablet_manager.cc:1398] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:03:57.921200 5274 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.921960 5433 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.922361 5433 raft_consensus.cc:686] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:57.923147 5433 consensus_queue.cc:237] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.927918 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:57.928442 5200 leader_election.cc:302] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:03:57.928774 5309 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 0.
I20240503 14:03:57.930284 5432 raft_consensus.cc:2781] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:57.930806 5432 raft_consensus.cc:482] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:57.931128 5432 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.932312 5462 raft_consensus.cc:2726] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.934816 5383 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 1.
I20240503 14:03:57.940065 5432 raft_consensus.cc:504] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.940404 5436 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.943531 5436 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:57.944672 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:57.944996 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:57.945330 5309 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.945638 5308 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.944584 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:57.948248 5234 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.952749 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:57.953514 5383 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.954944 5309 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 1.
I20240503 14:03:57.955118 5308 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 1.
I20240503 14:03:57.956123 5200 leader_election.cc:302] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.956408 5349 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:57.957319 5436 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.943245 5130 catalog_manager.cc:5484] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:57.959561 5436 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:57.962760 5234 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 1.
I20240503 14:03:57.962463 5436 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.964886 5487 raft_consensus.cc:2781] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:57.968062 5433 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:57.968655 5433 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.969214 5487 raft_consensus.cc:686] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:57.970161 5487 consensus_queue.cc:237] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:57.971979 5433 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.973520 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:57.977345 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:03:57.978260 5234 raft_consensus.cc:2379] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3ea46436f134d77b24e334796af1c56 in current term 1: Already voted for candidate d25004c4b8f84302b323f34a58117752 in this term.
I20240503 14:03:57.979377 5274 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:03:57.979564 5432 leader_election.cc:288] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:57.982370 5383 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 1.
I20240503 14:03:57.983019 5433 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:57.990099 5433 raft_consensus.cc:2726] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:03:57.993896 5130 catalog_manager.cc:5484] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 1 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:57.994218 5128 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 0 to 1, leader changed from <none> to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:03:58.097285 5436 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.115299 5436 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.111162 5433 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240503 14:03:58.141626 5432 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:03:58.142271 5432 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.144546 5432 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.147500 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:03:58.148402 5308 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 0.
I20240503 14:03:58.149838 5200 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.150789 5432 raft_consensus.cc:2781] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:03:58.151186 5432 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:03:58.151520 5432 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.154242 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:03:58.158026 5467 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.162058 5383 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 0.
I20240503 14:03:58.165447 5432 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.170066 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.170840 5383 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.176509 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.177233 5309 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:03:58.177939 5432 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.178396 5383 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 1.
I20240503 14:03:58.182135 5200 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:03:58.183194 5487 raft_consensus.cc:2781] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:03:58.183710 5487 raft_consensus.cc:686] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.184281 5309 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 1.
I20240503 14:03:58.184764 5487 consensus_queue.cc:237] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.188257 5467 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.189965 5454 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.202376 5128 catalog_manager.cc:5484] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 1 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.206995 5467 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.207986 5466 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.234894 5462 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:03:58.262583 5467 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.369010 5467 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.379717 5466 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.437382 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:41810
I20240503 14:03:58.438002 5309 raft_consensus.cc:595] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:03:58.443792 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:41810
I20240503 14:03:58.444350 5309 raft_consensus.cc:595] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:03:58.458328 5383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:59650
I20240503 14:03:58.458930 5383 raft_consensus.cc:595] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:03:58.464882 5383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:59650
I20240503 14:03:58.465411 5383 raft_consensus.cc:595] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:03:58.470188 5383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:59650
I20240503 14:03:58.470705 5383 raft_consensus.cc:595] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:03:58.479722 5487 consensus_queue.cc:1035] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.487222 5234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:37290
I20240503 14:03:58.487852 5234 raft_consensus.cc:595] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:03:58.497398 5462 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.500077 5234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:37290
I20240503 14:03:58.500849 5234 raft_consensus.cc:595] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:03:58.511607 5487 raft_consensus.cc:979] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.512122 5487 raft_consensus.cc:1067] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.522617 5307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41800
I20240503 14:03:58.523653 5307 raft_consensus.cc:482] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.524046 5307 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.528858 5432 consensus_queue.cc:1035] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:03:58.533821 5462 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.535745 5308 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:41810
I20240503 14:03:58.536464 5308 raft_consensus.cc:595] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:03:58.537380 5307 raft_consensus.cc:504] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.546521 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:58.547750 5233 raft_consensus.cc:3032] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.548316 5233 raft_consensus.cc:729] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.549597 5233 consensus_queue.cc:260] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.550311 5233 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.555168 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.556015 5383 raft_consensus.cc:3037] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.558773 5307 leader_election.cc:288] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.565533 5383 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:03:58.569478 5275 leader_election.cc:302] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.570749 5466 raft_consensus.cc:2781] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.571290 5466 raft_consensus.cc:686] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.572973 5466 consensus_queue.cc:237] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.581610 5233 raft_consensus.cc:2454] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:03:58.581248 5130 catalog_manager.cc:5484] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 1 to 2, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.587452 5436 raft_consensus.cc:979] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.588424 5462 raft_consensus.cc:1067] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.591496 5234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37274
I20240503 14:03:58.592310 5234 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.592792 5234 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.600268 5467 raft_consensus.cc:979] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:03:58.600826 5456 raft_consensus.cc:1067] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:03:58.603082 5383 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59638
I20240503 14:03:58.603204 5234 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.603742 5383 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.604195 5383 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.606009 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.606498 5234 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.606616 5307 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.608573 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.609185 5382 raft_consensus.cc:3032] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.609725 5382 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:58.610927 5382 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.612241 5382 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.612973 5307 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.614362 5200 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.615160 5487 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.615622 5487 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.616457 5487 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.618963 5383 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.621788 5383 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 2 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:03:58.622469 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.623217 5307 raft_consensus.cc:3032] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.623694 5307 raft_consensus.cc:729] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.624554 5307 consensus_queue.cc:260] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.626062 5307 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.626201 5129 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 1 to 2, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 2 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.627480 5382 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.626125 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:58.628815 5234 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.633738 5307 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 2.
I20240503 14:03:58.634967 5349 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.635234 5234 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 2.
I20240503 14:03:58.635771 5436 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.636328 5436 raft_consensus.cc:686] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:58.637095 5436 consensus_queue.cc:237] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.643527 5129 catalog_manager.cc:5484] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 1 to 2, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 2 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:03:58.659905 5487 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.665506 5454 raft_consensus.cc:979] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.665980 5454 raft_consensus.cc:1067] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.670768 5307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41798
I20240503 14:03:58.671406 5307 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.671783 5307 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.674149 5487 raft_consensus.cc:979] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.674588 5487 raft_consensus.cc:1067] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:03:58.678335 5454 raft_consensus.cc:979] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.678776 5454 raft_consensus.cc:1067] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.680565 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41800
I20240503 14:03:58.681170 5309 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.681532 5309 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.686652 5487 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:03:58.694484 5234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37274
I20240503 14:03:58.695721 5234 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.696321 5234 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.696682 5307 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.698483 5309 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.704375 5309 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.705396 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.706720 5307 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.706645 5456 raft_consensus.cc:979] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.707183 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.708333 5456 raft_consensus.cc:1067] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.708492 5381 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.707821 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:58.708930 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:03:58.710167 5233 raft_consensus.cc:3032] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.710506 5233 raft_consensus.cc:729] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.708909 5381 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:58.711251 5233 consensus_queue.cc:260] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.709642 5232 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.711392 5381 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.712759 5233 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.713212 5381 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.708642 5234 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.717511 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:03:58.718250 5231 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.718760 5231 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.721788 5234 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.723315 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.726526 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.727171 5383 raft_consensus.cc:3032] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.727517 5383 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:03:58.728389 5383 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.729341 5381 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:03:58.729820 5383 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.724792 5309 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.732771 5275 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.733835 5456 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.734324 5456 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.735105 5456 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.738137 5232 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:03:58.738968 5231 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.740974 5231 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.743038 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.743624 5381 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.744488 5233 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:03:58.746063 5274 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters: d25004c4b8f84302b323f34a58117752
I20240503 14:03:58.747175 5467 raft_consensus.cc:2781] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.747718 5467 raft_consensus.cc:686] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.742041 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.748787 5467 consensus_queue.cc:237] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.751391 5307 raft_consensus.cc:3032] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.751886 5307 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.752799 5307 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.754015 5307 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.756462 5128 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 1 to 2, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.753942 5383 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.756336 5381 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.770975 5200 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:03:58.771301 5309 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.772349 5432 raft_consensus.cc:2781] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.772848 5432 raft_consensus.cc:686] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.773864 5307 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.773612 5432 consensus_queue.cc:237] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.773209 5129 catalog_manager.cc:5484] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 1 to 2, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.775159 5200 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:03:58.776057 5487 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.776530 5487 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.777307 5487 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.781097 5128 catalog_manager.cc:5484] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 1 to 2, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 2 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.789337 5456 raft_consensus.cc:979] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.789737 5467 raft_consensus.cc:1067] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:03:58.791209 5233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:03:58.791680 5233 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:03:58.791991 5233 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.798061 5233 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.798256 5129 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 1 to 2, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 2 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:58.801127 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:03:58.801373 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:03:58.802053 5383 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.801787 5307 raft_consensus.cc:3032] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:03:58.804534 5307 raft_consensus.cc:729] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:03:58.805413 5307 consensus_queue.cc:260] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.806581 5307 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:58.811069 5383 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.812412 5200 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:03:58.800307 5233 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:03:58.813334 5487 raft_consensus.cc:2781] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:03:58.815186 5487 raft_consensus.cc:686] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:03:58.814497 5307 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:03:58.816159 5487 consensus_queue.cc:237] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:03:58.825090 5129 catalog_manager.cc:5484] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 1 to 2, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 2 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:03:59.081964 5233 raft_consensus.cc:1261] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.083277 5436 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.090799 5307 raft_consensus.cc:1261] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.092479 5462 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:03:59.146936 5233 raft_consensus.cc:1261] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.148569 5456 consensus_queue.cc:1035] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.157495 5383 raft_consensus.cc:1261] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.159047 5456 consensus_queue.cc:1035] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.181596 5383 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.182777 5487 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.193445 5307 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.195351 5517 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:03:59.197643 5383 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.199558 5517 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240503 14:03:59.209020 5383 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.212231 5467 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:03:59.219264 5307 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.221168 5517 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:03:59.226876 5233 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.230341 5433 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:03:59.238432 5381 raft_consensus.cc:1261] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.241173 5487 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.249752 5307 raft_consensus.cc:1261] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.251749 5488 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.258292 5309 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.259696 5488 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.271878 5381 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.273614 5487 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.280246 5383 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:03:59.286201 5383 raft_consensus.cc:1261] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.287364 5433 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:03:59.294114 5233 raft_consensus.cc:1261] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:03:59.295481 5456 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:01.568244 5099 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240503 14:04:01.569079 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:01.590338 5233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:37034
I20240503 14:04:01.590896 5233 raft_consensus.cc:595] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:04:01.601524 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:01.602097 5309 raft_consensus.cc:595] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:04:01.603183 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:04:01.603683 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:01.675132 5466 raft_consensus.cc:979] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:01.675575 5527 raft_consensus.cc:1067] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:01.677109 5383 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59638
I20240503 14:04:01.677562 5383 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:01.677850 5383 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:01.682375 5383 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:01.683858 5383 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:01.684999 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:01.685000 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:01.685871 5309 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:01.685875 5233 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:01.687604 5309 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:01.688460 5309 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:01.689450 5309 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:01.691300 5233 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 3.
I20240503 14:04:01.692421 5348 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:04:01.693248 5462 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:01.694461 5309 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 3.
I20240503 14:04:01.694574 5462 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:01.695505 5462 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:01.702183 5129 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 2 to 3, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 3 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:02.049531 5487 raft_consensus.cc:979] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:02.050043 5534 raft_consensus.cc:1067] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:02.051724 5383 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59624
I20240503 14:04:02.052346 5383 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:02.052635 5383 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:02.057055 5383 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:02.058459 5383 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:02.059451 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:02.059464 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:02.060323 5309 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:02.060344 5233 raft_consensus.cc:3032] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:02.060915 5233 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:02.061733 5233 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:02.063050 5233 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:02.067848 5309 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 3.
I20240503 14:04:02.069300 5349 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:02.070096 5233 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 3.
I20240503 14:04:02.070206 5462 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:02.070756 5462 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:02.071511 5462 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:02.078778 5129 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 2 to 3, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 3 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:02.242882 5233 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:02.244388 5462 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:02.253788 5309 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:02.255360 5528 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:02.540408 5309 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:02.541574 5462 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:02.550757 5231 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:02.552740 5544 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:02.604609 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:02.609733 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:02.610164 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:03.610977 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:03.616199 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:03.616636 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:04.617642 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:04.623829 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:04.624296 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:05.625171 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:05.630316 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:05.630913 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:06.631807 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:06.638211 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:06.638598 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:07.639645 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:07.644922 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:07.645368 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:08.646164 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:08.651598 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:08.651998 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:09.652858 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:09.658499 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:09.659273 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:10.660178 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:10.664800 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:10.665199 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:11.689307 5099 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240503 14:04:11.693902 5231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:37290
I20240503 14:04:11.694603 5231 raft_consensus.cc:595] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:04:11.699767 5231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:37290
I20240503 14:04:11.700359 5231 raft_consensus.cc:595] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:04:11.705058 5383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:59650
I20240503 14:04:11.705567 5383 raft_consensus.cc:595] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:04:11.710335 5231 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:37290
I20240503 14:04:11.710850 5231 raft_consensus.cc:595] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:04:11.715390 5383 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
mode: GRACEFUL
new_leader_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:59650
I20240503 14:04:11.715893 5383 raft_consensus.cc:595] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Received request to transfer leadership to TS d3ea46436f134d77b24e334796af1c56
I20240503 14:04:11.805356 5552 raft_consensus.cc:979] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.805850 5555 raft_consensus.cc:1067] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.807446 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41798
I20240503 14:04:11.807926 5309 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:11.808334 5309 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:11.813625 5309 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.815454 5309 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 4 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:11.816476 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:11.816465 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:11.817322 5383 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:11.817350 5231 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:11.817809 5383 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:11.818670 5383 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.819994 5383 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:11.824892 5231 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 4.
I20240503 14:04:11.826321 5274 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:11.826628 5383 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 4.
I20240503 14:04:11.828019 5541 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:11.828511 5541 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:11.829991 5541 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.837684 5129 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 3 to 4, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 4 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:11.890472 5534 raft_consensus.cc:979] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.890878 5534 raft_consensus.cc:1067] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.892262 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41800
I20240503 14:04:11.892747 5309 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:11.893057 5309 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.897719 5309 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.899262 5309 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:11.900218 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:11.900166 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:11.900962 5383 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.900998 5231 raft_consensus.cc:3032] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:11.901536 5231 raft_consensus.cc:729] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:11.902364 5231 consensus_queue.cc:260] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.903723 5231 raft_consensus.cc:3037] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.907944 5383 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:11.909509 5275 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:11.910462 5541 raft_consensus.cc:2781] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:11.910584 5231 raft_consensus.cc:2454] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:11.911044 5541 raft_consensus.cc:686] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:11.912031 5541 consensus_queue.cc:237] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.920011 5129 catalog_manager.cc:5484] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 2 to 3, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 3 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:11.951488 5561 raft_consensus.cc:979] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.951977 5534 raft_consensus.cc:1067] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.953536 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41800
I20240503 14:04:11.954042 5309 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:11.954314 5309 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.958983 5309 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.960542 5309 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:11.961369 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:11.961494 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:11.962036 5231 raft_consensus.cc:3032] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:11.962078 5383 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.962448 5231 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:11.963182 5231 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.964208 5231 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:11.967291 5383 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:11.968371 5275 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:11.969182 5541 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:11.969602 5541 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:11.969985 5231 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:11.970556 5541 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:11.977420 5129 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 2 to 3, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 3 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:11.995294 5555 raft_consensus.cc:979] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.995739 5552 raft_consensus.cc:1067] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:11.997166 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41798
I20240503 14:04:11.997642 5309 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:11.997911 5309 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:12.002315 5309 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.003820 5309 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 4 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:12.004664 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:12.004868 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:12.005198 5231 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:12.005409 5381 raft_consensus.cc:3032] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:12.005728 5381 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:12.006378 5381 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.007323 5381 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:12.009867 5231 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 4.
I20240503 14:04:12.010824 5274 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:12.011480 5541 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:12.011826 5541 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:12.012323 5381 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 4.
I20240503 14:04:12.012593 5541 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.019635 5129 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 3 to 4, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 4 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:12.134828 5561 raft_consensus.cc:979] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: : Instructing follower d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:12.135257 5534 raft_consensus.cc:1067] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Signalling peer d3ea46436f134d77b24e334796af1c56 to start an election
I20240503 14:04:12.136570 5309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
from {username='slave'} at 127.0.0.1:41800
I20240503 14:04:12.137040 5309 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:12.137308 5309 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:12.141801 5309 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.143280 5309 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:12.144061 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:12.144335 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:12.144685 5231 raft_consensus.cc:3032] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:12.144853 5381 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:12.145061 5231 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:12.145887 5231 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.147197 5231 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:12.150153 5381 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:12.151175 5275 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:12.151775 5231 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 3.
I20240503 14:04:12.151876 5541 raft_consensus.cc:2781] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:12.152487 5541 raft_consensus.cc:686] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:12.153596 5541 consensus_queue.cc:237] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:12.160035 5129 catalog_manager.cc:5484] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 2 to 3, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 3 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:12.218668 5381 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:12.220067 5541 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:12.231997 5231 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:12.234256 5568 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:04:12.352305 5381 raft_consensus.cc:1261] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.353546 5568 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:12.362599 5231 raft_consensus.cc:1261] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.364944 5554 consensus_queue.cc:1035] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:12.490021 5381 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.491254 5554 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:12.500555 5231 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.502422 5554 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:12.554108 5231 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:12.555315 5554 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:12.563952 5381 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:12.566160 5541 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:12.714550 5231 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.715796 5541 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:12.725996 5381 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:12.727226 5568 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:14.755770 5099 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240503 14:04:14.756817 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:14.763520 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:14.764233 5309 raft_consensus.cc:595] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:14.766503 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:14.766925 5309 raft_consensus.cc:595] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:04:14.768893 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:14.769328 5309 raft_consensus.cc:595] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:04:14.771196 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:14.771615 5309 raft_consensus.cc:595] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:14.773496 5309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:14.773892 5309 raft_consensus.cc:595] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:14.774823 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240503 14:04:14.775233 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:14.802543 5541 raft_consensus.cc:979] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:14.803062 5581 raft_consensus.cc:1067] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:14.804555 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:04:14.805022 5231 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:14.805367 5231 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:14.811640 5231 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:14.813635 5231 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:14.814738 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:14.814955 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:14.815428 5381 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:14.815495 5309 raft_consensus.cc:3032] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:14.815793 5309 raft_consensus.cc:729] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:14.816381 5309 consensus_queue.cc:260] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:14.817289 5309 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:14.820246 5381 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:04:14.821399 5200 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:04:14.822292 5309 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 2.
I20240503 14:04:14.822911 5582 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:14.823815 5582 raft_consensus.cc:686] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:14.824613 5582 consensus_queue.cc:237] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:14.831558 5128 catalog_manager.cc:5484] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 1 to 2, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 2 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:14.968529 5541 raft_consensus.cc:979] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:14.969018 5581 raft_consensus.cc:1067] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:14.970376 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:04:14.970829 5231 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:14.971074 5231 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:14.986377 5231 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:14.987861 5231 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:14.988737 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:14.988835 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:14.989212 5309 raft_consensus.cc:3032] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:14.989334 5381 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:14.989531 5309 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:14.990233 5309 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:14.991174 5309 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:15.003584 5381 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 4.
I20240503 14:04:15.003597 5309 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 4.
I20240503 14:04:15.005028 5200 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:15.005723 5582 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:15.006219 5582 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:15.007177 5582 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.014320 5128 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 3 to 4, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 4 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:15.051187 5541 raft_consensus.cc:979] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:15.051599 5581 raft_consensus.cc:1067] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:15.052979 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:04:15.053450 5231 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:15.053771 5231 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.081277 5231 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.083101 5231 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 5 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:15.084028 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:15.084211 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:15.084565 5309 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:04:15.084682 5381 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.084858 5309 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:15.085537 5309 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.086563 5309 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.107606 5381 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 5.
I20240503 14:04:15.107606 5309 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 5.
I20240503 14:04:15.108978 5200 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:15.109607 5582 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:04:15.110000 5582 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:15.110885 5582 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.117684 5128 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 4 to 5, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 5 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:15.131434 5541 raft_consensus.cc:979] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:15.131879 5581 raft_consensus.cc:1067] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:15.133297 5381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "22097a93651744deb19d0f1a634b5817"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59638
I20240503 14:04:15.133832 5381 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:15.134124 5381 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:15.154389 5381 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.156412 5381 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:15.157312 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:15.157497 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:15.157955 5231 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:15.158068 5309 raft_consensus.cc:3032] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:15.158399 5309 raft_consensus.cc:729] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:15.159039 5309 consensus_queue.cc:260] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.159940 5309 raft_consensus.cc:3037] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:15.167750 5541 raft_consensus.cc:979] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:15.168761 5581 raft_consensus.cc:1067] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:15.170521 5381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59638
I20240503 14:04:15.170991 5381 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:15.171404 5381 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.186139 5309 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 4.
I20240503 14:04:15.186455 5381 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.186136 5231 raft_consensus.cc:2454] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 4.
I20240503 14:04:15.187678 5349 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:15.188720 5381 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 5 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:15.188791 5552 raft_consensus.cc:2781] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:15.190992 5552 raft_consensus.cc:686] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 4 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:15.190786 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:15.191749 5233 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.192287 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:15.192142 5552 consensus_queue.cc:237] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.192898 5309 raft_consensus.cc:3032] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:04:15.193243 5309 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:15.193923 5309 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.195078 5309 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:15.201776 5128 catalog_manager.cc:5484] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 3 to 4, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 4 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:15.222955 5307 raft_consensus.cc:1261] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:15.224282 5582 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:15.224478 5309 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 5.
I20240503 14:04:15.224471 5233 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 5.
I20240503 14:04:15.225777 5349 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:15.226776 5552 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:04:15.227267 5552 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:15.228298 5552 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:15.237057 5129 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 4 to 5, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 5 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:15.237876 5381 raft_consensus.cc:1261] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:15.239550 5597 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:04:15.424949 5309 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:15.425004 5381 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:15.426177 5582 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:15.426854 5597 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:15.590626 5309 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:15.591940 5582 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:04:15.601264 5381 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:15.602926 5582 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:04:15.635821 5233 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:15.637116 5552 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:04:15.647444 5309 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:15.649643 5592 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:04:15.665313 5309 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:15.666604 5592 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:15.678565 5233 raft_consensus.cc:1261] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:15.680147 5552 consensus_queue.cc:1035] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:04:15.776270 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:15.781262 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:15.781697 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:16.782509 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:16.787354 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:16.787791 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:04:18.142613 5199 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.000s sys 0.010s
W20240503 14:04:18.143182 5199 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.531s user 0.000s sys 0.011s
I20240503 14:04:18.750236 5616 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:18.751127 5616 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.754221 5616 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:18.921847 5617 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:18.922628 5617 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.924937 5617 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
W20240503 14:04:18.961390 5255 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea060 after lost signal to thread 5102
I20240503 14:04:18.977871 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:18.978703 5382 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 5.
I20240503 14:04:18.979286 5618 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:18.980042 5618 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.932046 5616 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:18.985922 5619 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:18.986137 5616 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.988509 5620 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d3ea46436f134d77b24e334796af1c56)
I20240503 14:04:18.989162 5620 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.987346 5618 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:18.989666 5616 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:18.992055 5620 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:18.992792 5275 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:18.993769 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:18.994180 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:18.994268 5621 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3ea46436f134d77b24e334796af1c56)
I20240503 14:04:18.995864 5620 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:18.996186 5621 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.997042 5620 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.998858 5621 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:18.988515 5619 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:18.999853 5383 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 4.
I20240503 14:04:19.001225 5619 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:19.003316 5616 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240503 14:04:19.003806 5616 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:19.004817 5616 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.007216 5622 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:19.007874 5622 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.010776 5622 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.012255 5622 raft_consensus.cc:482] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d3ea46436f134d77b24e334796af1c56)
I20240503 14:04:19.012833 5622 raft_consensus.cc:504] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.013715 5620 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.014866 5622 leader_election.cc:288] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.015038 5616 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.020884 5552 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:19.021479 5552 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.024366 5552 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:19.024955 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.028677 5616 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.029808 5275 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:19.030247 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.030864 5381 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.030913 5613 raft_consensus.cc:482] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d3ea46436f134d77b24e334796af1c56)
I20240503 14:04:19.031000 5581 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240503 14:04:19.031925 5581 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:19.031653 5613 raft_consensus.cc:504] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.032367 5581 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:19.033967 5613 leader_election.cc:288] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.034679 5613 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:19.035033 5618 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
I20240503 14:04:19.035243 5613 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.035519 5618 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
W20240503 14:04:19.037698 5255 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea740 after lost signal to thread 5165
I20240503 14:04:19.037779 5618 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.038426 5381 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 6.
W20240503 14:04:19.038986 5255 debug-util.cc:398] Leaking SignalData structure 0x7b08001d99e0 after lost signal to thread 5258
I20240503 14:04:19.042035 5581 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.042580 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.043731 5275 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:19.044123 5380 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:04:19.044525 5616 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:04:19.045089 5275 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
W20240503 14:04:19.045300 5255 debug-util.cc:398] Leaking SignalData structure 0x7b080014f1e0 after lost signal to thread 5333
I20240503 14:04:19.046011 5618 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:04:19.046386 5618 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 71d66a9bdfcc42479aa03d5b7fd25484)
W20240503 14:04:19.046384 5255 debug-util.cc:398] Leaking SignalData structure 0x7b080008f0e0 after lost signal to thread 5407
I20240503 14:04:19.046921 5618 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.046437 5616 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:19.051630 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.051956 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.052636 5613 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.053958 5308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.055462 5624 raft_consensus.cc:482] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:19.057897 5381 raft_consensus.cc:1226] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Rejecting Update request from peer 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 5. Current term is 6. Ops: []
I20240503 14:04:19.059492 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.058125 5307 raft_consensus.cc:2379] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d25004c4b8f84302b323f34a58117752 in current term 5: Already voted for candidate d3ea46436f134d77b24e334796af1c56 in this term.
I20240503 14:04:19.060761 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.058607 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.060817 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.061390 5616 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.062219 5234 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 5.
I20240503 14:04:19.062316 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:19.062543 5381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.062858 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.063910 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.064433 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.065539 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.062147 5233 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 2.
I20240503 14:04:19.067036 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.066984 5234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.067620 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.063367 5624 raft_consensus.cc:504] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.068495 5230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.070132 5624 leader_election.cc:288] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.070161 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.071799 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.071916 5581 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.073194 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.073156 5305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.074223 5382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:19.074616 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.074849 5348 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.075616 5552 raft_consensus.cc:2726] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 14:04:19.075340 5309 raft_consensus.cc:1226] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Rejecting Update request from peer 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 4. Current term is 5. Ops: []
I20240503 14:04:19.077373 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a2a31688a4a699253667f00b716b2" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.078241 5613 consensus_queue.cc:1046] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:04:19.078924 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c08089db36f41f19299ca62b01986ec" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.079754 5613 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:04:19.080310 5613 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:19.080372 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:19.081485 5613 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.081831 5349 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.082546 5618 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.084123 5308 raft_consensus.cc:2379] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d25004c4b8f84302b323f34a58117752 in current term 3: Already voted for candidate d3ea46436f134d77b24e334796af1c56 in this term.
I20240503 14:04:19.083366 5621 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:19.085330 5618 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.087980 5200 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.088804 5306 raft_consensus.cc:1226] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Rejecting Update request from peer 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 2. Current term is 3. Ops: []
I20240503 14:04:19.089175 5623 raft_consensus.cc:2726] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.090446 5200 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.091055 5274 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters: d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.091768 5623 raft_consensus.cc:2726] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240503 14:04:19.093226 5200 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.095239 5624 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240503 14:04:19.095647 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.097198 5307 raft_consensus.cc:1226] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Rejecting Update request from peer 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 5. Current term is 6. Ops: []
I20240503 14:04:19.095702 5624 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:19.098361 5624 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.103475 5621 raft_consensus.cc:2726] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240503 14:04:19.104429 5623 raft_consensus.cc:2726] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.105171 5623 consensus_queue.cc:1046] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:04:19.105077 5128 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 5 to 6, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 6 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:19.106259 5623 raft_consensus.cc:3032] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:04:19.106540 5623 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:19.107221 5623 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.108829 5274 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters: d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.109467 5623 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:04:19.110105 5617 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:04:19.110625 5617 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:19.111137 5617 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.111179 5349 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d25004c4b8f84302b323f34a58117752; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.112839 5621 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.112424 5613 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.105173 5200 leader_election.cc:302] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.115249 5622 raft_consensus.cc:2726] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.116550 5200 leader_election.cc:302] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.119230 5622 raft_consensus.cc:2726] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240503 14:04:19.120729 5200 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484; no voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56
I20240503 14:04:19.122993 5622 raft_consensus.cc:2726] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240503 14:04:19.123620 5620 consensus_queue.cc:1046] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:19.124900 5620 raft_consensus.cc:3032] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:19.125265 5230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:19.125540 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:19.124729 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22097a93651744deb19d0f1a634b5817" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:19.125345 5620 raft_consensus.cc:729] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:19.127040 5274 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.128329 5274 leader_election.cc:302] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.128101 5620 consensus_queue.cc:260] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.127969 5581 raft_consensus.cc:2726] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.129202 5616 raft_consensus.cc:2726] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240503 14:04:19.129165 5624 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.131858 5620 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.133369 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.136351 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:19.138362 5274 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.139278 5616 raft_consensus.cc:2726] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:04:19.132221 5624 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 6 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.150609 5621 raft_consensus.cc:2726] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.152675 5231 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 5.
I20240503 14:04:19.154937 5274 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters: d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.155866 5617 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.156469 5624 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:04:19.157769 5624 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:19.159839 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.160243 5624 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.161871 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:19.165087 5274 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752
I20240503 14:04:19.165382 5617 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 3 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.166251 5616 raft_consensus.cc:2726] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 14:04:19.181027 5128 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 4 to 5, leader changed from 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 5 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:19.331792 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:19.338642 5307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:19.339493 5307 raft_consensus.cc:595] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:19.342602 5307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:19.343422 5307 raft_consensus.cc:595] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Received request to transfer leadership to TS 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:19.344795 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:04:19.345373 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:19.470260 5380 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 14:04:19.471855 5624 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:04:19.486187 5231 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 14:04:19.489287 5624 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:04:19.547286 5307 raft_consensus.cc:1226] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Rejecting Update request from peer d25004c4b8f84302b323f34a58117752 for earlier term 5. Current term is 6. Ops: []
I20240503 14:04:19.548691 5621 consensus_queue.cc:1046] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:04:19.549583 5552 raft_consensus.cc:3032] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:04:19.549964 5552 raft_consensus.cc:729] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:19.550740 5552 consensus_queue.cc:260] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.552018 5552 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.576226 5307 raft_consensus.cc:1226] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Rejecting Update request from peer d25004c4b8f84302b323f34a58117752 for earlier term 2. Current term is 3. Ops: []
I20240503 14:04:19.577602 5552 consensus_queue.cc:1046] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:19.578548 5621 raft_consensus.cc:3032] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:19.578982 5621 raft_consensus.cc:729] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:19.579988 5621 consensus_queue.cc:260] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.581730 5621 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.602834 5617 raft_consensus.cc:979] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:19.603463 5624 raft_consensus.cc:1067] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:19.605255 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:04:19.605862 5231 raft_consensus.cc:482] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:19.606212 5231 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.612884 5231 raft_consensus.cc:504] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.614838 5231 leader_election.cc:288] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.616369 5307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:19.616276 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c570d2dbff30430290e8f782ad2ea0d1" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.617120 5307 raft_consensus.cc:3032] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:19.617201 5380 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.617630 5307 raft_consensus.cc:729] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:19.618511 5307 consensus_queue.cc:260] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.619786 5307 raft_consensus.cc:3037] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:19.626325 5380 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 3.
I20240503 14:04:19.626325 5307 raft_consensus.cc:2454] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 3.
I20240503 14:04:19.627658 5200 leader_election.cc:302] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:04:19.628782 5623 raft_consensus.cc:2781] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:19.629267 5623 raft_consensus.cc:686] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:19.630117 5623 consensus_queue.cc:237] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.637914 5128 catalog_manager.cc:5484] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 2 to 3, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 3 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:19.707485 5380 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:19.708923 5624 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:04:19.722265 5231 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:04:19.723850 5624 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:04:19.732472 5616 raft_consensus.cc:979] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: : Instructing follower 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:19.733040 5624 raft_consensus.cc:1067] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 LEADER]: Signalling peer 71d66a9bdfcc42479aa03d5b7fd25484 to start an election
I20240503 14:04:19.734725 5231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb"
dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
from {username='slave'} at 127.0.0.1:37282
I20240503 14:04:19.735373 5231 raft_consensus.cc:482] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:19.735780 5231 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.742283 5231 raft_consensus.cc:504] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.744998 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:19.745615 5306 raft_consensus.cc:3032] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:04:19.745990 5306 raft_consensus.cc:729] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:19.746913 5306 consensus_queue.cc:260] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.748275 5306 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.748862 5231 leader_election.cc:288] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 6 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:19.750222 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4454a646fb5a4246abe5215f17d099bb" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:19.751175 5380 raft_consensus.cc:3037] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:04:19.757269 5306 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 6.
I20240503 14:04:19.758548 5200 leader_election.cc:302] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:19.759207 5380 raft_consensus.cc:2454] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 6.
I20240503 14:04:19.759338 5623 raft_consensus.cc:2781] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:04:19.759941 5623 raft_consensus.cc:686] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:19.760865 5623 consensus_queue.cc:237] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:19.768023 5129 catalog_manager.cc:5484] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 5 to 6, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193). New cstate: current_term: 6 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:20.101788 5380 raft_consensus.cc:1261] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:20.103206 5623 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:20.116576 5306 raft_consensus.cc:1261] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:20.118356 5623 consensus_queue.cc:1035] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:20.229821 5380 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 14:04:20.231467 5623 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:04:20.242671 5306 raft_consensus.cc:1261] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240503 14:04:20.244488 5620 consensus_queue.cc:1035] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:04:20.346395 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:20.353986 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:20.354534 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:20.716470 5623 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:20.717043 5623 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.719135 5623 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:20.720036 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:20.720036 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:20.720789 5306 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 3.
I20240503 14:04:20.720901 5380 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 3.
I20240503 14:04:20.721827 5200 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:20.722432 5623 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 14:04:20.722812 5623 raft_consensus.cc:482] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:20.723150 5623 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:20.730137 5623 raft_consensus.cc:504] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.732343 5623 leader_election.cc:288] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 election: Requested vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:20.733100 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:20.733322 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e7042a091e45c8b75cbbc88c950e67" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:20.733640 5306 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:20.734004 5380 raft_consensus.cc:3037] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:20.738768 5306 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 4.
I20240503 14:04:20.740016 5200 leader_election.cc:302] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:20.741107 5623 raft_consensus.cc:2781] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:20.741608 5623 raft_consensus.cc:686] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 LEADER]: Becoming Leader. State: Replica: 71d66a9bdfcc42479aa03d5b7fd25484, State: Running, Role: LEADER
I20240503 14:04:20.741995 5380 raft_consensus.cc:2454] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d66a9bdfcc42479aa03d5b7fd25484 in term 4.
I20240503 14:04:20.742563 5623 consensus_queue.cc:237] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.750345 5129 catalog_manager.cc:5484] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:20.838984 5552 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:20.839596 5552 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.841796 5552 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:20.842607 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:20.843092 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:20.843856 5306 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 3.
I20240503 14:04:20.845300 5349 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:20.846418 5552 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240503 14:04:20.846863 5552 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:20.847208 5552 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:20.856232 5552 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.858861 5552 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:20.859553 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:20.860203 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:20.860965 5306 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:20.869482 5306 raft_consensus.cc:2454] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 4.
I20240503 14:04:20.870967 5349 leader_election.cc:302] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters: 71d66a9bdfcc42479aa03d5b7fd25484
I20240503 14:04:20.871942 5552 raft_consensus.cc:2781] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:20.873178 5552 raft_consensus.cc:686] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 4 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:20.874100 5552 consensus_queue.cc:237] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.883246 5129 catalog_manager.cc:5484] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:20.942813 5616 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:20.943341 5616 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.945499 5616 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:20.946353 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" is_pre_election: true
I20240503 14:04:20.946554 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:20.947067 5231 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 5.
I20240503 14:04:20.947221 5380 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 6.
I20240503 14:04:20.948171 5274 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:20.949015 5616 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240503 14:04:20.949362 5616 raft_consensus.cc:482] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:20.949687 5616 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:04:20.955562 5616 raft_consensus.cc:504] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.957561 5616 leader_election.cc:288] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 7 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:20.958467 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:20.958722 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11308338f3fe4e4a9aef449bbd587c34" candidate_uuid: "d3ea46436f134d77b24e334796af1c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752"
I20240503 14:04:20.959115 5231 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 5 FOLLOWER]: Advancing to term 7
I20240503 14:04:20.959321 5380 raft_consensus.cc:3037] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:04:20.965802 5380 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 7.
I20240503 14:04:20.966110 5231 raft_consensus.cc:2454] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ea46436f134d77b24e334796af1c56 in term 7.
I20240503 14:04:20.966975 5275 leader_election.cc:302] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d25004c4b8f84302b323f34a58117752, d3ea46436f134d77b24e334796af1c56; no voters:
I20240503 14:04:20.968047 5616 raft_consensus.cc:2781] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 14:04:20.969808 5616 raft_consensus.cc:686] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 7 LEADER]: Becoming Leader. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:20.970801 5616 consensus_queue.cc:237] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:20.979763 5129 catalog_manager.cc:5484] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 reported cstate change: term changed from 5 to 7, leader changed from d25004c4b8f84302b323f34a58117752 (127.4.250.195) to d3ea46436f134d77b24e334796af1c56 (127.4.250.194). New cstate: current_term: 7 leader_uuid: "d3ea46436f134d77b24e334796af1c56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:21.068199 5623 raft_consensus.cc:482] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d25004c4b8f84302b323f34a58117752)
I20240503 14:04:21.068936 5623 raft_consensus.cc:504] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:21.071677 5623 leader_election.cc:288] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355), d25004c4b8f84302b323f34a58117752 (127.4.250.195:42405)
I20240503 14:04:21.072518 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d3ea46436f134d77b24e334796af1c56" is_pre_election: true
I20240503 14:04:21.072785 5380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacb2e13659140d5a7b52329f9c578db" candidate_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d25004c4b8f84302b323f34a58117752" is_pre_election: true
I20240503 14:04:21.073678 5306 raft_consensus.cc:2354] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 3. Current term is 4.
I20240503 14:04:21.074887 5200 leader_election.cc:398] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Vote denied by peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355) with higher term. Message: Invalid argument: T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 3. Current term is 4.
I20240503 14:04:21.075126 5200 leader_election.cc:401] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240503 14:04:21.075752 5623 raft_consensus.cc:3037] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Advancing to term 4
I20240503 14:04:21.084518 5623 raft_consensus.cc:2726] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355) with higher term. Message: Invalid argument: T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71d66a9bdfcc42479aa03d5b7fd25484 for earlier term 3. Current term is 4.
I20240503 14:04:21.194314 5380 raft_consensus.cc:1261] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240503 14:04:21.195683 5623 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:21.213641 5306 raft_consensus.cc:1261] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Refusing update from remote peer 71d66a9bdfcc42479aa03d5b7fd25484: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240503 14:04:21.215360 5620 consensus_queue.cc:1035] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:21.355413 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:21.362011 5306 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d3ea46436f134d77b24e334796af1c56"
mode: GRACEFUL
new_leader_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:38662
I20240503 14:04:21.362566 5306 raft_consensus.cc:595] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Received request to transfer leadership to TS d25004c4b8f84302b323f34a58117752
I20240503 14:04:21.363629 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 14:04:21.364045 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:21.386027 5231 raft_consensus.cc:1261] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240503 14:04:21.387356 5552 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:21.397341 5306 raft_consensus.cc:1261] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240503 14:04:21.399093 5552 consensus_queue.cc:1035] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:21.535961 5616 raft_consensus.cc:979] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: : Instructing follower d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:21.536409 5648 raft_consensus.cc:1067] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Signalling peer d25004c4b8f84302b323f34a58117752 to start an election
I20240503 14:04:21.537770 5382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f"
dest_uuid: "d25004c4b8f84302b323f34a58117752"
from {username='slave'} at 127.0.0.1:59638
I20240503 14:04:21.538275 5382 raft_consensus.cc:482] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:21.538558 5382 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:04:21.543478 5382 raft_consensus.cc:504] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:21.545243 5382 leader_election.cc:288] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 7 election: Requested vote from peers 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935), d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355)
I20240503 14:04:21.546108 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "71d66a9bdfcc42479aa03d5b7fd25484"
I20240503 14:04:21.546744 5231 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:04:21.547938 5306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76eec11ab4f41a8a14915ab78fbdc2f" candidate_uuid: "d25004c4b8f84302b323f34a58117752" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d3ea46436f134d77b24e334796af1c56"
I20240503 14:04:21.548630 5306 raft_consensus.cc:3032] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:04:21.548979 5306 raft_consensus.cc:729] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d3ea46436f134d77b24e334796af1c56, State: Running, Role: LEADER
I20240503 14:04:21.549751 5306 consensus_queue.cc:260] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:21.551040 5306 raft_consensus.cc:3037] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:04:21.553542 5231 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 7.
I20240503 14:04:21.554625 5348 leader_election.cc:302] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d66a9bdfcc42479aa03d5b7fd25484, d25004c4b8f84302b323f34a58117752; no voters:
I20240503 14:04:21.555271 5657 raft_consensus.cc:2781] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Leader election won for term 7
I20240503 14:04:21.555745 5657 raft_consensus.cc:686] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 7 LEADER]: Becoming Leader. State: Replica: d25004c4b8f84302b323f34a58117752, State: Running, Role: LEADER
I20240503 14:04:21.556578 5657 consensus_queue.cc:237] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } }
I20240503 14:04:21.557343 5306 raft_consensus.cc:2454] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25004c4b8f84302b323f34a58117752 in term 7.
I20240503 14:04:21.563629 5129 catalog_manager.cc:5484] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 reported cstate change: term changed from 6 to 7, leader changed from d3ea46436f134d77b24e334796af1c56 (127.4.250.194) to d25004c4b8f84302b323f34a58117752 (127.4.250.195). New cstate: current_term: 7 leader_uuid: "d25004c4b8f84302b323f34a58117752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:21.571072 5382 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240503 14:04:21.572495 5231 raft_consensus.cc:1261] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Refusing update from remote peer d3ea46436f134d77b24e334796af1c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240503 14:04:21.572546 5648 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25004c4b8f84302b323f34a58117752" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:04:21.573983 5616 consensus_queue.cc:1035] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:04:21.953327 5306 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240503 14:04:21.954689 5667 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ea46436f134d77b24e334796af1c56" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:04:21.963922 5231 raft_consensus.cc:1261] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Refusing update from remote peer d25004c4b8f84302b323f34a58117752: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240503 14:04:21.965274 5657 consensus_queue.cc:1035] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d66a9bdfcc42479aa03d5b7fd25484" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:04:22.364951 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:22.370245 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:22.370668 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:23.371451 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:23.376488 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:23.376972 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:24.377802 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:24.383735 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:24.384182 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:25.385089 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:25.390372 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:25.390823 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:26.416759 5099 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240503 14:04:26.417408 5099 tablet_server.cc:178] TabletServer@127.4.250.193:0 shutting down...
I20240503 14:04:26.448931 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:26.450012 5099 tablet_replica.cc:330] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
W20240503 14:04:26.450350 5274 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 14:04:26.451337 5099 raft_consensus.cc:2229] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.452934 5099 raft_consensus.cc:2258] T c570d2dbff30430290e8f782ad2ea0d1 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:26.454138 5274 consensus_peers.cc:485] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 -> Peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Couldn't send request to peer 71d66a9bdfcc42479aa03d5b7fd25484. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:26.455991 5099 tablet_replica.cc:330] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.456635 5099 raft_consensus.cc:2229] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.457123 5099 raft_consensus.cc:2258] T 11308338f3fe4e4a9aef449bbd587c34 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.459018 5099 tablet_replica.cc:330] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.459534 5099 raft_consensus.cc:2229] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.459961 5099 raft_consensus.cc:2258] T e09a2a31688a4a699253667f00b716b2 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.461763 5099 tablet_replica.cc:330] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.462241 5099 raft_consensus.cc:2229] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.462604 5099 raft_consensus.cc:2258] T 1c08089db36f41f19299ca62b01986ec P 71d66a9bdfcc42479aa03d5b7fd25484 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.464495 5099 tablet_replica.cc:330] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.465180 5099 raft_consensus.cc:2229] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.466008 5099 raft_consensus.cc:2258] T 4454a646fb5a4246abe5215f17d099bb P 71d66a9bdfcc42479aa03d5b7fd25484 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.467808 5099 tablet_replica.cc:330] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.468323 5099 raft_consensus.cc:2229] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.469121 5099 raft_consensus.cc:2258] T 03e7042a091e45c8b75cbbc88c950e67 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.470762 5099 tablet_replica.cc:330] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.471213 5099 raft_consensus.cc:2229] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.471606 5099 raft_consensus.cc:2258] T 22097a93651744deb19d0f1a634b5817 P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.473322 5099 tablet_replica.cc:330] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.473729 5099 raft_consensus.cc:2229] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.474126 5099 raft_consensus.cc:2258] T aacb2e13659140d5a7b52329f9c578db P 71d66a9bdfcc42479aa03d5b7fd25484 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.475802 5099 tablet_replica.cc:330] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484: stopping tablet replica
I20240503 14:04:26.476264 5099 raft_consensus.cc:2229] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.476739 5099 raft_consensus.cc:2258] T a76eec11ab4f41a8a14915ab78fbdc2f P 71d66a9bdfcc42479aa03d5b7fd25484 [term 7 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:26.568995 5348 consensus_peers.cc:485] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 -> Peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Couldn't send request to peer 71d66a9bdfcc42479aa03d5b7fd25484. Status: Network error: Client connection negotiation failed: client connection to 127.4.250.193:43935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:26.583951 5099 tablet_server.cc:194] TabletServer@127.4.250.193:0 shutdown complete.
I20240503 14:04:26.601054 5099 tablet_server.cc:178] TabletServer@127.4.250.194:0 shutting down...
I20240503 14:04:26.629601 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:26.630456 5099 tablet_replica.cc:330] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.631127 5099 raft_consensus.cc:2229] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.631803 5099 raft_consensus.cc:2258] T 4454a646fb5a4246abe5215f17d099bb P d3ea46436f134d77b24e334796af1c56 [term 6 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:26.633517 5349 consensus_peers.cc:485] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 -> Peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Couldn't send request to peer d3ea46436f134d77b24e334796af1c56. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:26.634142 5099 tablet_replica.cc:330] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.634662 5099 raft_consensus.cc:2229] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.635048 5099 raft_consensus.cc:2258] T a76eec11ab4f41a8a14915ab78fbdc2f P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.636893 5099 tablet_replica.cc:330] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.637451 5099 raft_consensus.cc:2229] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.637925 5099 raft_consensus.cc:2258] T aacb2e13659140d5a7b52329f9c578db P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.639562 5099 tablet_replica.cc:330] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.640067 5099 raft_consensus.cc:2229] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.640857 5099 raft_consensus.cc:2258] T e09a2a31688a4a699253667f00b716b2 P d3ea46436f134d77b24e334796af1c56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.642524 5099 tablet_replica.cc:330] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.643043 5099 raft_consensus.cc:2229] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 7 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.643720 5099 raft_consensus.cc:2258] T 11308338f3fe4e4a9aef449bbd587c34 P d3ea46436f134d77b24e334796af1c56 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.645365 5099 tablet_replica.cc:330] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.645781 5099 raft_consensus.cc:2229] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.646189 5099 raft_consensus.cc:2258] T 03e7042a091e45c8b75cbbc88c950e67 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.647696 5099 tablet_replica.cc:330] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.648169 5099 raft_consensus.cc:2229] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.648532 5099 raft_consensus.cc:2258] T 22097a93651744deb19d0f1a634b5817 P d3ea46436f134d77b24e334796af1c56 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.650063 5099 tablet_replica.cc:330] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.650457 5099 raft_consensus.cc:2229] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.650826 5099 raft_consensus.cc:2258] T c570d2dbff30430290e8f782ad2ea0d1 P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.652508 5099 tablet_replica.cc:330] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56: stopping tablet replica
I20240503 14:04:26.652937 5099 raft_consensus.cc:2229] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.653661 5099 raft_consensus.cc:2258] T 1c08089db36f41f19299ca62b01986ec P d3ea46436f134d77b24e334796af1c56 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:26.670672 5349 consensus_peers.cc:485] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 -> Peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Couldn't send request to peer d3ea46436f134d77b24e334796af1c56. Status: Network error: Client connection negotiation failed: client connection to 127.4.250.194:40355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:04:26.680956 5349 consensus_peers.cc:485] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 -> Peer d3ea46436f134d77b24e334796af1c56 (127.4.250.194:40355): Couldn't send request to peer d3ea46436f134d77b24e334796af1c56. Status: Network error: Client connection negotiation failed: client connection to 127.4.250.194:40355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:26.682232 5099 tablet_server.cc:194] TabletServer@127.4.250.194:0 shutdown complete.
I20240503 14:04:26.699386 5099 tablet_server.cc:178] TabletServer@127.4.250.195:0 shutting down...
I20240503 14:04:26.728462 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:26.729135 5099 tablet_replica.cc:330] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.729852 5099 raft_consensus.cc:2229] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.730423 5099 raft_consensus.cc:2258] T c570d2dbff30430290e8f782ad2ea0d1 P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.732934 5099 tablet_replica.cc:330] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.733670 5099 raft_consensus.cc:2229] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 4 LEADER]: Raft consensus shutting down.
W20240503 14:04:26.734182 5348 consensus_peers.cc:485] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 -> Peer 71d66a9bdfcc42479aa03d5b7fd25484 (127.4.250.193:43935): Couldn't send request to peer 71d66a9bdfcc42479aa03d5b7fd25484. Status: Network error: Client connection negotiation failed: client connection to 127.4.250.193:43935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:26.734661 5099 raft_consensus.cc:2258] T aacb2e13659140d5a7b52329f9c578db P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.736476 5099 tablet_replica.cc:330] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.737040 5099 raft_consensus.cc:2229] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.737485 5099 raft_consensus.cc:2258] T 03e7042a091e45c8b75cbbc88c950e67 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.739135 5099 tablet_replica.cc:330] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.739670 5099 raft_consensus.cc:2229] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.740474 5099 raft_consensus.cc:2258] T 22097a93651744deb19d0f1a634b5817 P d25004c4b8f84302b323f34a58117752 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.742223 5099 tablet_replica.cc:330] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.742753 5099 raft_consensus.cc:2229] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.743185 5099 raft_consensus.cc:2258] T 11308338f3fe4e4a9aef449bbd587c34 P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.744906 5099 tablet_replica.cc:330] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.745409 5099 raft_consensus.cc:2229] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.745851 5099 raft_consensus.cc:2258] T e09a2a31688a4a699253667f00b716b2 P d25004c4b8f84302b323f34a58117752 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.747539 5099 tablet_replica.cc:330] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.748029 5099 raft_consensus.cc:2229] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 7 LEADER]: Raft consensus shutting down.
I20240503 14:04:26.748859 5099 raft_consensus.cc:2258] T a76eec11ab4f41a8a14915ab78fbdc2f P d25004c4b8f84302b323f34a58117752 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.750517 5099 tablet_replica.cc:330] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.751008 5099 raft_consensus.cc:2229] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.751456 5099 raft_consensus.cc:2258] T 1c08089db36f41f19299ca62b01986ec P d25004c4b8f84302b323f34a58117752 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.753216 5099 tablet_replica.cc:330] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752: stopping tablet replica
I20240503 14:04:26.753696 5099 raft_consensus.cc:2229] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:26.754194 5099 raft_consensus.cc:2258] T 4454a646fb5a4246abe5215f17d099bb P d25004c4b8f84302b323f34a58117752 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:26.781626 5099 tablet_server.cc:194] TabletServer@127.4.250.195:0 shutdown complete.
I20240503 14:04:26.798724 5099 master.cc:537] Master@127.4.250.254:38841 shutting down...
I20240503 14:04:26.999107 5186 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20240503 14:04:27.010087 5099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:27.010699 5099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:27.011034 5099 tablet_replica.cc:330] T 00000000000000000000000000000000 P e7509361dc6d4b9988670406b4ecd2f4: stopping tablet replica
I20240503 14:04:27.029328 5099 master.cc:552] Master@127.4.250.254:38841 shutdown complete.
71d66a9bdfcc42479aa03d5b7fd25484 4
d25004c4b8f84302b323f34a58117752 1
d3ea46436f134d77b24e334796af1c56 4
71d66a9bdfcc42479aa03d5b7fd25484 3
d25004c4b8f84302b323f34a58117752 3
d3ea46436f134d77b24e334796af1c56 3
71d66a9bdfcc42479aa03d5b7fd25484 0
d25004c4b8f84302b323f34a58117752 1
d3ea46436f134d77b24e334796af1c56 8
71d66a9bdfcc42479aa03d5b7fd25484 3
d25004c4b8f84302b323f34a58117752 3
d3ea46436f134d77b24e334796af1c56 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32715 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240503 14:04:27.069257 5099 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.250.254:46567
I20240503 14:04:27.070374 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:27.076186 5679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.076541 5680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.077520 5682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:27.077608 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:27.078788 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:27.078989 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:27.079150 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745067079130 us; error 0 us; skew 500 ppm
I20240503 14:04:27.079677 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:27.082365 5099 webserver.cc:460] Webserver started at http://127.4.250.254:37681/ using document root <none> and password file <none>
I20240503 14:04:27.082872 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:27.083050 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:27.083336 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:27.084625 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/master-0-root/instance:
uuid: "fc7ff289dd6e460c986a1f75cac4e208"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.089234 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:04:27.092693 5687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.093488 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240503 14:04:27.093873 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/master-0-root
uuid: "fc7ff289dd6e460c986a1f75cac4e208"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.094172 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:27.108794 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:27.110025 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:27.148136 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.254:46567
I20240503 14:04:27.148275 5738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.254:46567 every 8 connection(s)
I20240503 14:04:27.152000 5739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.161634 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208: Bootstrap starting.
I20240503 14:04:27.165930 5739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.170099 5739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208: No bootstrap required, opened a new log
I20240503 14:04:27.172245 5739 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER }
I20240503 14:04:27.172717 5739 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.172951 5739 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7ff289dd6e460c986a1f75cac4e208, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.173489 5739 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER }
I20240503 14:04:27.173949 5739 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:27.174175 5739 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:27.174441 5739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:27.178480 5739 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER }
I20240503 14:04:27.179019 5739 leader_election.cc:302] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7ff289dd6e460c986a1f75cac4e208; no voters:
I20240503 14:04:27.180096 5739 leader_election.cc:288] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:27.180442 5742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:27.181777 5742 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 1 LEADER]: Becoming Leader. State: Replica: fc7ff289dd6e460c986a1f75cac4e208, State: Running, Role: LEADER
I20240503 14:04:27.182432 5742 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER }
I20240503 14:04:27.182940 5739 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:27.187120 5744 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7ff289dd6e460c986a1f75cac4e208. Latest consensus state: current_term: 1 leader_uuid: "fc7ff289dd6e460c986a1f75cac4e208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER } }
I20240503 14:04:27.187325 5743 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7ff289dd6e460c986a1f75cac4e208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7ff289dd6e460c986a1f75cac4e208" member_type: VOTER } }
I20240503 14:04:27.187776 5744 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:27.187939 5743 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:27.191005 5749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:27.196166 5749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:27.198541 5099 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:27.204672 5749 catalog_manager.cc:1347] Generated new cluster ID: ff50ebf2dd5a4cba8bc341fe5da83f40
I20240503 14:04:27.204957 5749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:27.223616 5749 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:27.224833 5749 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:27.241370 5749 catalog_manager.cc:5857] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208: Generated new TSK 0
I20240503 14:04:27.241953 5749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:27.266302 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:27.272053 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.273250 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.274377 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:27.274639 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:27.275487 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:27.275681 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:27.275803 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745067275791 us; error 0 us; skew 500 ppm
I20240503 14:04:27.276348 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:27.282879 5099 webserver.cc:460] Webserver started at http://127.4.250.193:41475/ using document root <none> and password file <none>
I20240503 14:04:27.283378 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:27.283566 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:27.283835 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:27.284994 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-0-root/instance:
uuid: "9f7a820b09714c96aec13be58ab289ab"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.289489 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:27.292733 5768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.293546 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:04:27.293843 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
uuid: "9f7a820b09714c96aec13be58ab289ab"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.294137 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:27.306119 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:27.307278 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:27.308729 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:27.310945 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:27.311142 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.311368 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:27.311550 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.349066 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.193:43397
I20240503 14:04:27.349143 5830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.193:43397 every 8 connection(s)
I20240503 14:04:27.353528 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:27.360761 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.361840 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.365299 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:27.366535 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:27.367216 5831 heartbeater.cc:347] Connected to a master server at 127.4.250.254:46567
I20240503 14:04:27.367568 5831 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:27.367913 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:27.368160 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:27.368332 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745067368310 us; error 0 us; skew 500 ppm
I20240503 14:04:27.368326 5831 heartbeater.cc:511] Master 127.4.250.254:46567 requested a full tablet report, sending...
I20240503 14:04:27.369060 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:27.370716 5704 ts_manager.cc:194] Registered new tserver with Master: 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397)
I20240503 14:04:27.371847 5099 webserver.cc:460] Webserver started at http://127.4.250.194:37885/ using document root <none> and password file <none>
I20240503 14:04:27.372434 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:27.372629 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:27.372757 5704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49106
I20240503 14:04:27.372875 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:27.374222 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-1-root/instance:
uuid: "79ba0c2d1c934cf291e699f8e9e70400"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.378719 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:04:27.382166 5843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.383014 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:04:27.383320 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
uuid: "79ba0c2d1c934cf291e699f8e9e70400"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.383626 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:27.406101 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:27.407347 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:27.408855 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:27.411258 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:27.411504 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.411736 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:27.411891 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.475450 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.194:46853
I20240503 14:04:27.475565 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.194:46853 every 8 connection(s)
I20240503 14:04:27.480227 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:27.487107 5909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:27.488997 5910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:27.492774 5906 heartbeater.cc:347] Connected to a master server at 127.4.250.254:46567
I20240503 14:04:27.493467 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:27.493626 5906 heartbeater.cc:464] Registering TS with master...
W20240503 14:04:27.494431 5912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:27.494715 5906 heartbeater.cc:511] Master 127.4.250.254:46567 requested a full tablet report, sending...
I20240503 14:04:27.495409 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:27.495636 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:27.495817 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745067495800 us; error 0 us; skew 500 ppm
I20240503 14:04:27.496604 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:27.497071 5704 ts_manager.cc:194] Registered new tserver with Master: 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:27.498864 5704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49122
I20240503 14:04:27.499573 5099 webserver.cc:460] Webserver started at http://127.4.250.195:43771/ using document root <none> and password file <none>
I20240503 14:04:27.500288 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:27.500509 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:27.500782 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:27.501987 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-2-root/instance:
uuid: "913051779cda40fc82f6279197502e7b"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.506477 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:27.509804 5917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.510545 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:04:27.510836 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
uuid: "913051779cda40fc82f6279197502e7b"
format_stamp: "Formatted at 2024-05-03 14:04:27 on dist-test-slave-8wrt"
I20240503 14:04:27.511116 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714745034190430-5099-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:27.526685 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:27.527877 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:27.529358 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:27.531641 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:27.531865 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.532141 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:27.532315 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:27.572288 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.195:45795
I20240503 14:04:27.572372 5979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.195:45795 every 8 connection(s)
I20240503 14:04:27.585830 5980 heartbeater.cc:347] Connected to a master server at 127.4.250.254:46567
I20240503 14:04:27.586232 5980 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:27.586961 5980 heartbeater.cc:511] Master 127.4.250.254:46567 requested a full tablet report, sending...
I20240503 14:04:27.588899 5704 ts_manager.cc:194] Registered new tserver with Master: 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:27.589047 5099 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01377718s
I20240503 14:04:27.590266 5704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49124
I20240503 14:04:27.591617 5099 test_util.cc:268] Using random seed: 849500074
I20240503 14:04:27.615403 5704 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49138:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:04:27.618407 5704 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:04:27.691085 5945 tablet_service.cc:1467] Processing CreateTablet for tablet c254be31c226480ba71d7231d79ee234 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 14:04:27.693492 5944 tablet_service.cc:1467] Processing CreateTablet for tablet 7629315cc19b4c15a9f1e725df1dcb0b (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 14:04:27.697468 5942 tablet_service.cc:1467] Processing CreateTablet for tablet 47b05f61b04c4c948c9229d79112f8d3 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 14:04:27.699582 5941 tablet_service.cc:1467] Processing CreateTablet for tablet ed43220acdc744b6ac9c98e97bb91ba7 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 14:04:27.701776 5940 tablet_service.cc:1467] Processing CreateTablet for tablet a3ec7afab9254f5fa4db7ef9bd18aec9 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 14:04:27.703959 5939 tablet_service.cc:1467] Processing CreateTablet for tablet 762e19a422db4495a8d7cd33ccb7e483 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 14:04:27.695811 5943 tablet_service.cc:1467] Processing CreateTablet for tablet 7bce6196cffa46549a33cf717d7a3638 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 14:04:27.706312 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c254be31c226480ba71d7231d79ee234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.710280 5938 tablet_service.cc:1467] Processing CreateTablet for tablet 2e46ad74f5054c388807beb37a70fd8f (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 14:04:27.711898 5939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762e19a422db4495a8d7cd33ccb7e483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.717283 5870 tablet_service.cc:1467] Processing CreateTablet for tablet 7629315cc19b4c15a9f1e725df1dcb0b (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 14:04:27.718235 5871 tablet_service.cc:1467] Processing CreateTablet for tablet c254be31c226480ba71d7231d79ee234 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 14:04:27.718770 5870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7629315cc19b4c15a9f1e725df1dcb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.719610 5871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c254be31c226480ba71d7231d79ee234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.721215 5869 tablet_service.cc:1467] Processing CreateTablet for tablet 7bce6196cffa46549a33cf717d7a3638 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 14:04:27.722647 5869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bce6196cffa46549a33cf717d7a3638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.717171 5937 tablet_service.cc:1467] Processing CreateTablet for tablet 93fd44cd62ff467fa8b2cc596459e721 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 14:04:27.725366 5936 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf18ff07174497a9554a9fcb513af3d (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 14:04:27.726320 5868 tablet_service.cc:1467] Processing CreateTablet for tablet 47b05f61b04c4c948c9229d79112f8d3 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 14:04:27.727458 5865 tablet_service.cc:1467] Processing CreateTablet for tablet 762e19a422db4495a8d7cd33ccb7e483 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 14:04:27.729166 5867 tablet_service.cc:1467] Processing CreateTablet for tablet ed43220acdc744b6ac9c98e97bb91ba7 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 14:04:27.730366 5862 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf18ff07174497a9554a9fcb513af3d (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 14:04:27.730697 5793 tablet_service.cc:1467] Processing CreateTablet for tablet 47b05f61b04c4c948c9229d79112f8d3 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240503 14:04:27.731595 5866 tablet_service.cc:1467] Processing CreateTablet for tablet a3ec7afab9254f5fa4db7ef9bd18aec9 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 14:04:27.732188 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b05f61b04c4c948c9229d79112f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.732790 5796 tablet_service.cc:1467] Processing CreateTablet for tablet c254be31c226480ba71d7231d79ee234 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240503 14:04:27.734182 5796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c254be31c226480ba71d7231d79ee234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.734112 5794 tablet_service.cc:1467] Processing CreateTablet for tablet 7bce6196cffa46549a33cf717d7a3638 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240503 14:04:27.736742 5792 tablet_service.cc:1467] Processing CreateTablet for tablet ed43220acdc744b6ac9c98e97bb91ba7 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240503 14:04:27.737437 5795 tablet_service.cc:1467] Processing CreateTablet for tablet 7629315cc19b4c15a9f1e725df1dcb0b (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240503 14:04:27.727887 5864 tablet_service.cc:1467] Processing CreateTablet for tablet 2e46ad74f5054c388807beb37a70fd8f (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 14:04:27.739240 5791 tablet_service.cc:1467] Processing CreateTablet for tablet a3ec7afab9254f5fa4db7ef9bd18aec9 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240503 14:04:27.729180 5863 tablet_service.cc:1467] Processing CreateTablet for tablet 93fd44cd62ff467fa8b2cc596459e721 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 14:04:27.741535 5790 tablet_service.cc:1467] Processing CreateTablet for tablet 762e19a422db4495a8d7cd33ccb7e483 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240503 14:04:27.743710 5789 tablet_service.cc:1467] Processing CreateTablet for tablet 2e46ad74f5054c388807beb37a70fd8f (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240503 14:04:27.745831 5788 tablet_service.cc:1467] Processing CreateTablet for tablet 93fd44cd62ff467fa8b2cc596459e721 (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240503 14:04:27.747988 5787 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf18ff07174497a9554a9fcb513af3d (DEFAULT_TABLE table=test-workload [id=73d0b6de71c34a3b9a42961f9a1afa49]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240503 14:04:27.749626 5941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43220acdc744b6ac9c98e97bb91ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.752985 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ec7afab9254f5fa4db7ef9bd18aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.756458 5942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b05f61b04c4c948c9229d79112f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.757264 5944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7629315cc19b4c15a9f1e725df1dcb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.762327 5943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bce6196cffa46549a33cf717d7a3638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.762995 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e46ad74f5054c388807beb37a70fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.764606 5794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bce6196cffa46549a33cf717d7a3638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.768498 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ec7afab9254f5fa4db7ef9bd18aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.772500 5790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762e19a422db4495a8d7cd33ccb7e483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.773272 5936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf18ff07174497a9554a9fcb513af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.727854 5868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b05f61b04c4c948c9229d79112f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.774214 5864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e46ad74f5054c388807beb37a70fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.775342 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e46ad74f5054c388807beb37a70fd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.776424 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fd44cd62ff467fa8b2cc596459e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.776424 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7629315cc19b4c15a9f1e725df1dcb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.780467 5863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fd44cd62ff467fa8b2cc596459e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.781002 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf18ff07174497a9554a9fcb513af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.783516 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3ec7afab9254f5fa4db7ef9bd18aec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.784142 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43220acdc744b6ac9c98e97bb91ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.788497 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43220acdc744b6ac9c98e97bb91ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.790647 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93fd44cd62ff467fa8b2cc596459e721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.795104 5865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762e19a422db4495a8d7cd33ccb7e483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.796013 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf18ff07174497a9554a9fcb513af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:27.842815 5996 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:27.859628 5997 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:27.861557 5996 tablet_bootstrap.cc:654] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.870301 5995 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:27.872159 5997 tablet_bootstrap.cc:654] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.906137 5996 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:27.907582 5996 ts_tablet_manager.cc:1367] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.065s user 0.016s sys 0.003s
I20240503 14:04:27.909739 5996 raft_consensus.cc:348] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.910306 5996 raft_consensus.cc:374] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.910593 5996 raft_consensus.cc:729] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.911290 5996 consensus_queue.cc:260] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.899966 5995 tablet_bootstrap.cc:654] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.915122 5997 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:27.915123 5996 ts_tablet_manager.cc:1398] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:04:27.915500 5831 heartbeater.cc:503] Master 127.4.250.254:46567 was elected leader, sending a full tablet report...
I20240503 14:04:27.915938 5997 ts_tablet_manager.cc:1367] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.012s
I20240503 14:04:27.916425 5996 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:27.918812 5997 raft_consensus.cc:348] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.919770 5997 raft_consensus.cc:374] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.920181 5997 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.921052 5997 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.922814 5996 tablet_bootstrap.cc:654] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.929239 5995 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:27.929776 5995 ts_tablet_manager.cc:1367] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.010s
I20240503 14:04:27.930913 5996 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:27.931493 5996 ts_tablet_manager.cc:1367] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240503 14:04:27.932693 5995 raft_consensus.cc:348] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.933538 5995 raft_consensus.cc:374] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.933849 5995 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.934502 5996 raft_consensus.cc:348] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.935410 5996 raft_consensus.cc:374] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.934701 5995 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.935741 5996 raft_consensus.cc:729] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.937253 5906 heartbeater.cc:503] Master 127.4.250.254:46567 was elected leader, sending a full tablet report...
I20240503 14:04:27.937882 5997 ts_tablet_manager.cc:1398] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.021s user 0.005s sys 0.015s
I20240503 14:04:27.937644 5996 consensus_queue.cc:260] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.938907 5997 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:27.940025 5996 ts_tablet_manager.cc:1398] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:04:27.940984 5996 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:27.945008 5997 tablet_bootstrap.cc:654] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.946895 5996 tablet_bootstrap.cc:654] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.956815 5995 ts_tablet_manager.cc:1398] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.027s user 0.007s sys 0.007s
I20240503 14:04:27.956998 5997 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:27.957702 5997 ts_tablet_manager.cc:1367] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20240503 14:04:27.957641 5980 heartbeater.cc:503] Master 127.4.250.254:46567 was elected leader, sending a full tablet report...
I20240503 14:04:27.957990 5995 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:27.956990 5996 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:27.959312 5996 ts_tablet_manager.cc:1367] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20240503 14:04:27.962054 5996 raft_consensus.cc:348] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.962849 5996 raft_consensus.cc:374] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.963153 5996 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.963896 5995 tablet_bootstrap.cc:654] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.963588 5997 raft_consensus.cc:348] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.963840 5996 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.964783 5997 raft_consensus.cc:374] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:27.965476 5997 raft_consensus.cc:729] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:27.966223 5996 ts_tablet_manager.cc:1398] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:04:27.966279 5997 consensus_queue.cc:260] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.967379 5996 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:27.968832 5997 ts_tablet_manager.cc:1398] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:04:27.969815 5997 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:27.974020 5996 tablet_bootstrap.cc:654] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:27.974568 6002 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:27.975077 6002 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.976023 5997 tablet_bootstrap.cc:654] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:27.980150 5907 tablet.cc:2369] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:04:27.983590 6000 raft_consensus.cc:482] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:27.984205 6000 raft_consensus.cc:504] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:27.987039 6000 leader_election.cc:288] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.000685 6002 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.030107 5997 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.029886 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.030684 5997 ts_tablet_manager.cc:1367] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.061s user 0.030s sys 0.028s
I20240503 14:04:28.030761 5806 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 0.
I20240503 14:04:28.030179 5996 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.031688 5996 ts_tablet_manager.cc:1367] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.065s user 0.036s sys 0.024s
I20240503 14:04:28.033027 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.033635 5997 raft_consensus.cc:348] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.034392 5997 raft_consensus.cc:374] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:04:28.034723 5847 leader_election.cc:341] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.034689 5997 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.035661 5846 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.034718 5996 raft_consensus.cc:348] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.036005 5997 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.036852 6002 raft_consensus.cc:2781] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.037357 5996 raft_consensus.cc:374] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.037391 6002 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.039359 5996 raft_consensus.cc:729] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.039582 6002 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.040228 5996 consensus_queue.cc:260] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.040957 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.042623 5996 ts_tablet_manager.cc:1398] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
W20240503 14:04:28.042693 5772 leader_election.cc:341] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.043783 5996 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.045152 5997 ts_tablet_manager.cc:1398] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240503 14:04:28.046064 5997 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.049820 5996 tablet_bootstrap.cc:654] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.051568 5997 tablet_bootstrap.cc:654] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.054905 6002 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.039054 5995 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.059860 5995 ts_tablet_manager.cc:1367] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.102s user 0.021s sys 0.003s
I20240503 14:04:28.060194 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.062343 5806 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.063280 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.061344 5996 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.061286 6000 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.063980 5881 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 0.
I20240503 14:04:28.064438 6000 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.065070 5996 ts_tablet_manager.cc:1367] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20240503 14:04:28.065109 5772 leader_election.cc:302] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.066385 6000 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.066916 6000 raft_consensus.cc:2781] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.067261 6000 raft_consensus.cc:482] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.067588 6000 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.068403 5996 raft_consensus.cc:348] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.069222 5996 raft_consensus.cc:374] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.061676 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:28.070705 5996 raft_consensus.cc:729] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.069397 5806 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 1.
I20240503 14:04:28.061910 5997 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
W20240503 14:04:28.071722 5847 leader_election.cc:341] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.071947 5997 ts_tablet_manager.cc:1367] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.000s
I20240503 14:04:28.071547 5996 consensus_queue.cc:260] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.074118 5996 ts_tablet_manager.cc:1398] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:04:28.074968 5997 raft_consensus.cc:348] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.075781 5997 raft_consensus.cc:374] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.075457 6000 raft_consensus.cc:504] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.076076 5997 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.064467 6002 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.064332 5995 raft_consensus.cc:348] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.076838 5997 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.077729 5995 raft_consensus.cc:374] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.078044 5995 raft_consensus.cc:729] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.078895 5996 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.079247 5997 ts_tablet_manager.cc:1398] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:04:28.079450 6000 leader_election.cc:288] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.080230 5997 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.080685 5995 consensus_queue.cc:260] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.085743 5996 tablet_bootstrap.cc:654] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.086609 5997 tablet_bootstrap.cc:654] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.095549 5997 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.096143 5997 ts_tablet_manager.cc:1367] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240503 14:04:28.096789 5995 ts_tablet_manager.cc:1398] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20240503 14:04:28.099015 5997 raft_consensus.cc:348] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.099898 5997 raft_consensus.cc:374] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.101648 5997 raft_consensus.cc:729] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.102562 5997 consensus_queue.cc:260] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.103116 5996 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.103724 5996 ts_tablet_manager.cc:1367] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.011s
I20240503 14:04:28.103960 5995 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.104903 5997 ts_tablet_manager.cc:1398] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:04:28.105881 5997 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.106812 5996 raft_consensus.cc:348] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.108904 5996 raft_consensus.cc:374] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:04:28.108840 5772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.194:46853, user_credentials={real_user=slave}} blocked reactor thread for 43788.8us
I20240503 14:04:28.109318 5996 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.110040 5996 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.110785 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.112581 5996 ts_tablet_manager.cc:1398] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:04:28.113548 5996 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.113555 5997 tablet_bootstrap.cc:654] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.115336 6004 raft_consensus.cc:482] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.115823 6004 raft_consensus.cc:504] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.120623 5996 tablet_bootstrap.cc:654] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.123183 5846 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.123636 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.124795 6014 raft_consensus.cc:2781] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.125301 6014 raft_consensus.cc:686] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:28.126134 6014 consensus_queue.cc:237] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.121882 6004 leader_election.cc:288] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397)
I20240503 14:04:28.128528 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.129588 5881 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.129829 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
W20240503 14:04:28.131076 5772 leader_election.cc:341] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.135591 6017 raft_consensus.cc:482] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.136142 6017 raft_consensus.cc:504] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.138192 5996 tablet_bootstrap.cc:492] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.138861 5996 ts_tablet_manager.cc:1367] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.006s
I20240503 14:04:28.140764 5881 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 1.
I20240503 14:04:28.141546 5995 tablet_bootstrap.cc:654] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:28.134369 5772 leader_election.cc:341] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.142284 5772 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f7a820b09714c96aec13be58ab289ab; no voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.143164 6000 raft_consensus.cc:2726] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:04:28.143623 5772 leader_election.cc:341] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.144981 5772 leader_election.cc:302] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.145466 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.145789 6000 raft_consensus.cc:2781] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.145499 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.141957 5996 raft_consensus.cc:348] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.147428 5880 raft_consensus.cc:2454] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 0.
I20240503 14:04:28.147708 6004 raft_consensus.cc:482] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.148272 6004 raft_consensus.cc:504] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.147584 5996 raft_consensus.cc:374] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.149333 5996 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.150282 6004 leader_election.cc:288] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.150064 5996 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.150509 5772 leader_election.cc:302] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:28.150024 5997 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.151945 5997 ts_tablet_manager.cc:1367] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.006s
I20240503 14:04:28.154654 5997 raft_consensus.cc:348] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.155391 5997 raft_consensus.cc:374] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.155678 5997 raft_consensus.cc:729] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.156397 5997 consensus_queue.cc:260] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.158887 5997 ts_tablet_manager.cc:1398] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:04:28.159822 5997 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.164790 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.164316 5702 catalog_manager.cc:5484] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
W20240503 14:04:28.166136 5921 leader_election.cc:341] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.166971 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.167654 5881 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.168757 5921 leader_election.cc:302] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:28.169673 6004 raft_consensus.cc:2781] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.170030 6004 raft_consensus.cc:482] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.170348 6004 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.174116 5997 tablet_bootstrap.cc:654] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.177490 6004 raft_consensus.cc:504] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.180866 5996 ts_tablet_manager.cc:1398] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.042s user 0.009s sys 0.013s
I20240503 14:04:28.181143 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.181807 5880 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.182103 5996 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.182904 6028 raft_consensus.cc:2781] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.185235 6028 raft_consensus.cc:482] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.186455 6028 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.138163 6017 leader_election.cc:288] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.186734 5997 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
W20240503 14:04:28.190215 5772 leader_election.cc:341] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.191536 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.191857 5880 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.192309 5806 raft_consensus.cc:2454] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.192909 5997 ts_tablet_manager.cc:1367] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s
I20240503 14:04:28.193425 5996 tablet_bootstrap.cc:654] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.193862 5921 leader_election.cc:302] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:28.189070 6004 leader_election.cc:288] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
W20240503 14:04:28.194646 5752 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:28.195603 6004 raft_consensus.cc:2781] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.195883 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.196648 5806 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.198211 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.196803 6004 raft_consensus.cc:686] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:28.198839 5806 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.198596 5997 raft_consensus.cc:348] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.199479 5920 leader_election.cc:302] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters: 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:28.203804 5997 raft_consensus.cc:374] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.204278 5997 raft_consensus.cc:729] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.205298 5997 consensus_queue.cc:260] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.204005 6028 raft_consensus.cc:504] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.207612 6031 raft_consensus.cc:2781] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.208021 6031 raft_consensus.cc:482] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.208370 6031 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.208683 5997 ts_tablet_manager.cc:1398] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240503 14:04:28.209177 6028 leader_election.cc:288] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.209690 5806 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.209815 5997 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.209931 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.212666 5880 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.213105 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
W20240503 14:04:28.215298 5772 leader_election.cc:341] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.201784 6000 raft_consensus.cc:686] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:28.216624 6000 consensus_queue.cc:237] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.217995 5997 tablet_bootstrap.cc:654] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.202575 6004 consensus_queue.cc:237] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.202481 5996 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.220898 5996 ts_tablet_manager.cc:1367] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.005s
I20240503 14:04:28.221606 6028 raft_consensus.cc:482] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.222126 6028 raft_consensus.cc:504] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.223414 5880 raft_consensus.cc:2454] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 1.
I20240503 14:04:28.223493 5996 raft_consensus.cc:348] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.224288 5996 raft_consensus.cc:374] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.224606 5996 raft_consensus.cc:729] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.224442 6028 leader_election.cc:288] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.225291 5996 consensus_queue.cc:260] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.225986 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.226665 5995 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.227416 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.228518 5881 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 0.
I20240503 14:04:28.229295 5995 ts_tablet_manager.cc:1367] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.126s user 0.023s sys 0.015s
I20240503 14:04:28.231058 5996 ts_tablet_manager.cc:1398] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:04:28.232192 5996 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.236966 5995 raft_consensus.cc:348] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.237723 5995 raft_consensus.cc:374] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.238014 5995 raft_consensus.cc:729] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.238209 5772 leader_election.cc:302] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.238689 5995 consensus_queue.cc:260] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.239578 6028 raft_consensus.cc:2781] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.240295 6028 raft_consensus.cc:686] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:28.241125 6028 consensus_queue.cc:237] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.241386 5703 catalog_manager.cc:5484] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.242000 5772 leader_election.cc:302] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:28.243168 6026 raft_consensus.cc:2781] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.243551 6026 raft_consensus.cc:482] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.243875 6026 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.248769 5995 ts_tablet_manager.cc:1398] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.019s user 0.005s sys 0.006s
I20240503 14:04:28.249648 5995 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.249244 6031 raft_consensus.cc:504] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.250885 6026 raft_consensus.cc:504] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.252240 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
W20240503 14:04:28.253924 5921 leader_election.cc:341] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:04:28.254503 5772 leader_election.cc:341] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.254832 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.256340 5881 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.257248 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:28.237530 5996 tablet_bootstrap.cc:654] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.255474 5995 tablet_bootstrap.cc:654] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.255821 6026 leader_election.cc:288] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
W20240503 14:04:28.259862 5772 leader_election.cc:341] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.262840 5881 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 1.
I20240503 14:04:28.264019 5772 leader_election.cc:302] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.265108 6026 raft_consensus.cc:2781] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.265569 6026 raft_consensus.cc:686] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:28.266347 6026 consensus_queue.cc:237] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.272521 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.273113 5806 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.274068 6031 leader_election.cc:288] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397)
I20240503 14:04:28.279124 5806 raft_consensus.cc:2454] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.280174 5920 leader_election.cc:302] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters: 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:28.280553 5995 tablet_bootstrap.cc:492] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.280946 6031 raft_consensus.cc:2781] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.281105 5995 ts_tablet_manager.cc:1367] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.007s
I20240503 14:04:28.281498 6031 raft_consensus.cc:686] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:28.272483 5996 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.282238 5996 ts_tablet_manager.cc:1367] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.004s
I20240503 14:04:28.282397 6031 consensus_queue.cc:237] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.284556 5996 raft_consensus.cc:348] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.285282 5996 raft_consensus.cc:374] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.284412 5995 raft_consensus.cc:348] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.287339 5995 raft_consensus.cc:374] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.287813 5995 raft_consensus.cc:729] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.288805 6014 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.285534 5996 raft_consensus.cc:729] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.289285 6014 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.289700 5996 consensus_queue.cc:260] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.291970 6014 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.293035 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.293736 5996 ts_tablet_manager.cc:1398] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
W20240503 14:04:28.294291 5847 leader_election.cc:341] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.294814 5996 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: Bootstrap starting.
I20240503 14:04:28.288595 5995 consensus_queue.cc:260] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.295532 5704 catalog_manager.cc:5484] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 1 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:28.297761 5995 ts_tablet_manager.cc:1398] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20240503 14:04:28.298767 5995 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.300405 5996 tablet_bootstrap.cc:654] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.295516 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.302060 5806 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 0.
I20240503 14:04:28.303146 5846 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.304260 6014 raft_consensus.cc:2781] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.304642 6014 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.304970 6014 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.310796 5995 tablet_bootstrap.cc:654] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.311983 5996 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: No bootstrap required, opened a new log
I20240503 14:04:28.312584 5996 ts_tablet_manager.cc:1367] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20240503 14:04:28.314124 6014 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.315639 5997 tablet_bootstrap.cc:492] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.316191 5997 ts_tablet_manager.cc:1367] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.107s user 0.051s sys 0.031s
I20240503 14:04:28.316826 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.317399 5806 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.318061 5996 raft_consensus.cc:348] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.318804 5996 raft_consensus.cc:374] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.319101 5996 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.319774 5996 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.320061 5997 raft_consensus.cc:348] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.320845 5997 raft_consensus.cc:374] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.321136 5997 raft_consensus.cc:729] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.322082 5996 ts_tablet_manager.cc:1398] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:04:28.321929 5997 consensus_queue.cc:260] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.322757 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:28.323539 6014 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
W20240503 14:04:28.323803 5847 leader_election.cc:341] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 913051779cda40fc82f6279197502e7b (127.4.250.195:45795): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:28.324124 5997 ts_tablet_manager.cc:1398] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:04:28.325129 5997 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.327037 5806 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 1.
I20240503 14:04:28.328171 5846 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters: 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.328867 6014 raft_consensus.cc:2781] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.331641 5997 tablet_bootstrap.cc:654] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.329522 6014 raft_consensus.cc:686] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:28.333448 6014 consensus_queue.cc:237] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.339318 5995 tablet_bootstrap.cc:492] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.339721 5995 ts_tablet_manager.cc:1367] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.008s
I20240503 14:04:28.340693 5701 catalog_manager.cc:5484] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.342593 5701 catalog_manager.cc:5484] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.341692 5995 raft_consensus.cc:348] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.345911 5995 raft_consensus.cc:374] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.346189 5995 raft_consensus.cc:729] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.346765 5995 consensus_queue.cc:260] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.348567 5995 ts_tablet_manager.cc:1398] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:04:28.349476 5995 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.350505 5997 tablet_bootstrap.cc:492] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.350858 5997 ts_tablet_manager.cc:1367] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.003s
I20240503 14:04:28.353190 5997 raft_consensus.cc:348] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.353905 5997 raft_consensus.cc:374] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.354162 5997 raft_consensus.cc:729] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.354900 5995 tablet_bootstrap.cc:654] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.354734 5997 consensus_queue.cc:260] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.354795 5703 catalog_manager.cc:5484] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 1 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.356947 5997 ts_tablet_manager.cc:1398] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240503 14:04:28.357777 5997 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: Bootstrap starting.
I20240503 14:04:28.362704 5997 tablet_bootstrap.cc:654] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.368566 5704 catalog_manager.cc:5484] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.386209 5997 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: No bootstrap required, opened a new log
I20240503 14:04:28.386332 5995 tablet_bootstrap.cc:492] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.386638 5997 ts_tablet_manager.cc:1367] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240503 14:04:28.386775 5995 ts_tablet_manager.cc:1367] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.015s
I20240503 14:04:28.388536 5997 raft_consensus.cc:348] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.389062 5997 raft_consensus.cc:374] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.388782 5995 raft_consensus.cc:348] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.389262 5997 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.389343 5995 raft_consensus.cc:374] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.389621 5995 raft_consensus.cc:729] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.389788 5997 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.390229 5995 consensus_queue.cc:260] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.391664 5997 ts_tablet_manager.cc:1398] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:04:28.392912 5995 ts_tablet_manager.cc:1398] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240503 14:04:28.393719 5995 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.399113 5995 tablet_bootstrap.cc:654] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.405047 5995 tablet_bootstrap.cc:492] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.405449 5995 ts_tablet_manager.cc:1367] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240503 14:04:28.407333 5995 raft_consensus.cc:348] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.407924 5995 raft_consensus.cc:374] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.408221 5995 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.408924 5995 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.410815 5995 ts_tablet_manager.cc:1398] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:04:28.411741 5995 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.416298 5995 tablet_bootstrap.cc:654] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.420753 5995 tablet_bootstrap.cc:492] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.421164 5995 ts_tablet_manager.cc:1367] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:04:28.423019 5995 raft_consensus.cc:348] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.423538 5995 raft_consensus.cc:374] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.423739 5995 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.424288 5995 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.425940 5995 ts_tablet_manager.cc:1398] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:04:28.426914 5995 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.430917 5995 tablet_bootstrap.cc:654] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.435215 5995 tablet_bootstrap.cc:492] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.435573 5995 ts_tablet_manager.cc:1367] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:04:28.437500 5995 raft_consensus.cc:348] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.437965 5995 raft_consensus.cc:374] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.438129 5995 raft_consensus.cc:729] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.438539 5995 consensus_queue.cc:260] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.440127 5995 ts_tablet_manager.cc:1398] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:04:28.440609 6031 raft_consensus.cc:482] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.441136 5995 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: Bootstrap starting.
I20240503 14:04:28.441001 6031 raft_consensus.cc:504] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.442826 6031 leader_election.cc:288] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.443574 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.444296 5806 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.444372 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.445056 5881 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.445904 5920 leader_election.cc:302] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:28.446695 5995 tablet_bootstrap.cc:654] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:28.446992 6031 raft_consensus.cc:2781] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.447352 6031 raft_consensus.cc:482] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.447700 6031 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.451480 5995 tablet_bootstrap.cc:492] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: No bootstrap required, opened a new log
I20240503 14:04:28.451952 5995 ts_tablet_manager.cc:1367] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240503 14:04:28.454274 5995 raft_consensus.cc:348] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.454236 6031 raft_consensus.cc:504] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.455099 5995 raft_consensus.cc:374] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:28.455391 5995 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Initialized, Role: FOLLOWER
I20240503 14:04:28.456063 5995 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.457603 6031 leader_election.cc:288] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.457521 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.457518 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.458441 5995 ts_tablet_manager.cc:1398] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240503 14:04:28.458478 5806 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.458933 5881 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.463606 5881 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.464681 5921 leader_election.cc:302] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:28.464871 5806 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.465565 6031 raft_consensus.cc:2781] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.466054 6031 raft_consensus.cc:686] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:28.466667 6031 consensus_queue.cc:237] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.474543 5701 catalog_manager.cc:5484] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 1 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:28.546350 6031 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.546805 6031 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.548455 6031 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.549228 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.549378 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.549880 5806 raft_consensus.cc:2379] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 913051779cda40fc82f6279197502e7b in current term 1: Already voted for candidate 79ba0c2d1c934cf291e699f8e9e70400 in this term.
I20240503 14:04:28.550693 5920 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 913051779cda40fc82f6279197502e7b; no voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.551314 6031 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.553957 6014 raft_consensus.cc:482] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.554373 6014 raft_consensus.cc:504] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.556038 6014 leader_election.cc:288] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.556219 6031 raft_consensus.cc:2726] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:28.557139 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fd44cd62ff467fa8b2cc596459e721" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.557097 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fd44cd62ff467fa8b2cc596459e721" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.557850 5806 raft_consensus.cc:2454] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 0.
I20240503 14:04:28.557945 5955 raft_consensus.cc:2454] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 0.
I20240503 14:04:28.558893 5847 leader_election.cc:302] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:28.559377 6014 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.559583 6002 raft_consensus.cc:2781] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.560024 6002 raft_consensus.cc:482] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.560415 6002 raft_consensus.cc:3037] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.565878 6002 raft_consensus.cc:504] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.567750 6002 leader_election.cc:288] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.568961 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fd44cd62ff467fa8b2cc596459e721" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.569209 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93fd44cd62ff467fa8b2cc596459e721" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:28.569584 5806 raft_consensus.cc:3037] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.569780 5955 raft_consensus.cc:3037] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.574496 6057 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:28.577709 5806 raft_consensus.cc:2454] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 1.
I20240503 14:04:28.578625 5846 leader_election.cc:302] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:28.579317 5955 raft_consensus.cc:2454] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 1.
I20240503 14:04:28.579478 6057 raft_consensus.cc:2781] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.580276 6057 raft_consensus.cc:686] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:28.581104 6057 consensus_queue.cc:237] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.583946 6037 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.584782 6037 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.588817 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.590500 6037 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.592674 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.594247 5805 raft_consensus.cc:2379] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 913051779cda40fc82f6279197502e7b in current term 1: Already voted for candidate 79ba0c2d1c934cf291e699f8e9e70400 in this term.
I20240503 14:04:28.595353 5920 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 913051779cda40fc82f6279197502e7b; no voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.597060 6037 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.601197 6026 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.605185 6037 raft_consensus.cc:2726] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:28.606489 6037 raft_consensus.cc:482] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.607158 6037 raft_consensus.cc:504] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.606609 5703 catalog_manager.cc:5484] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 1 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:28.610718 6037 leader_election.cc:288] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.613018 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.610184 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.615684 5921 leader_election.cc:302] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 913051779cda40fc82f6279197502e7b; no voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.616444 6037 raft_consensus.cc:2726] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:28.617803 5954 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.625474 6026 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.637681 6037 raft_consensus.cc:482] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.638211 6037 raft_consensus.cc:504] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.639997 6037 leader_election.cc:288] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.640673 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.640988 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400" is_pre_election: true
I20240503 14:04:28.641371 5805 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.641630 5880 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 0.
I20240503 14:04:28.642362 5920 leader_election.cc:302] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:28.643138 6031 raft_consensus.cc:2781] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:28.643424 6031 raft_consensus.cc:482] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:28.643666 6031 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.645164 6037 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.648931 6031 raft_consensus.cc:504] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.651937 5880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:28.652577 5880 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.655390 6037 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.660809 6026 consensus_queue.cc:1035] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:28.651530 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:28.662684 5805 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.662477 5880 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.650952 6031 leader_election.cc:288] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:28.667697 5954 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.668514 5921 leader_election.cc:302] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:28.669634 6031 raft_consensus.cc:2781] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:28.670212 6031 raft_consensus.cc:686] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:28.671136 6031 consensus_queue.cc:237] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.680717 5805 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 1.
I20240503 14:04:28.684348 6027 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:28.683423 5704 catalog_manager.cc:5484] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 1 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:28.688349 5954 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.698535 6027 consensus_queue.cc:1035] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.711694 6027 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.714555 6002 raft_consensus.cc:482] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:28.715073 6002 raft_consensus.cc:504] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:28.717089 6002 leader_election.cc:288] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913051779cda40fc82f6279197502e7b (127.4.250.195:45795), 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397)
I20240503 14:04:28.718636 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f7a820b09714c96aec13be58ab289ab" is_pre_election: true
I20240503 14:04:28.718873 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913051779cda40fc82f6279197502e7b" is_pre_election: true
I20240503 14:04:28.719578 5805 raft_consensus.cc:2379] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79ba0c2d1c934cf291e699f8e9e70400 in current term 1: Already voted for candidate 913051779cda40fc82f6279197502e7b in this term.
I20240503 14:04:28.720757 5846 leader_election.cc:302] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400; no voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.722023 6002 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:28.729120 6002 raft_consensus.cc:2726] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:28.798451 6004 consensus_queue.cc:1035] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.808609 6037 consensus_queue.cc:1035] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.819947 6002 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:28.832152 6002 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:28.881297 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c254be31c226480ba71d7231d79ee234"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:28.881896 5955 raft_consensus.cc:595] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.886736 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:28.887270 5955 raft_consensus.cc:595] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.902381 5881 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:53392
I20240503 14:04:28.902909 5881 raft_consensus.cc:595] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.907660 5881 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:53392
I20240503 14:04:28.908202 5881 raft_consensus.cc:595] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:28.912734 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:28.913220 5955 raft_consensus.cc:595] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Received request to transfer leadership to TS 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:28.917754 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:28.918270 5955 raft_consensus.cc:595] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Received request to transfer leadership to TS 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:28.930943 5805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:41534
I20240503 14:04:28.931494 5805 raft_consensus.cc:595] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Received request to transfer leadership to TS 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:28.938866 5805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:41534
I20240503 14:04:28.939427 5805 raft_consensus.cc:595] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Received request to transfer leadership to TS 913051779cda40fc82f6279197502e7b
I20240503 14:04:28.970903 6037 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.981060 6031 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:28.988832 6031 raft_consensus.cc:979] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: : Instructing follower 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:28.989333 6031 raft_consensus.cc:1067] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Signalling peer 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:28.990773 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:53390
I20240503 14:04:28.991322 5881 raft_consensus.cc:482] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:28.991649 5881 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:28.996995 5881 raft_consensus.cc:504] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:28.998646 5881 leader_election.cc:288] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:28.999763 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:28.999795 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:29.000597 5955 raft_consensus.cc:3032] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.000679 5805 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.000922 5955 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:29.001730 5955 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.002766 5955 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.006911 6002 consensus_queue.cc:1035] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:29.008669 5955 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.008685 5805 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.009927 5847 leader_election.cc:302] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:29.010754 6002 raft_consensus.cc:2781] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.011186 6002 raft_consensus.cc:686] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:29.012049 6002 consensus_queue.cc:237] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.020181 6014 consensus_queue.cc:1035] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:29.022334 5704 catalog_manager.cc:5484] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 1 to 2, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.096206 6084 raft_consensus.cc:979] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.096597 6014 raft_consensus.cc:1067] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.097967 5805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41514
I20240503 14:04:29.098460 5805 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.098728 5805 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.103266 5805 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.104859 5805 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:29.105767 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:29.105996 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.106369 5881 raft_consensus.cc:3032] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.106504 5954 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.106673 5881 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:29.107426 5881 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.108402 5881 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.111306 5954 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.112465 5772 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:29.113159 6027 raft_consensus.cc:2781] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.113322 5881 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.113643 6027 raft_consensus.cc:686] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.114497 6027 consensus_queue.cc:237] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.121546 5700 catalog_manager.cc:5484] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 1 to 2, leader changed from 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 2 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.148100 6031 consensus_queue.cc:1035] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:29.162303 6031 consensus_queue.cc:1035] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:04:29.169539 6027 raft_consensus.cc:979] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:29.170059 6017 raft_consensus.cc:1067] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Signalling peer 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:29.171465 6037 raft_consensus.cc:979] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.171967 6031 raft_consensus.cc:1067] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.171993 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:53384
I20240503 14:04:29.172758 5881 raft_consensus.cc:482] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.173185 5881 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.173656 5805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41520
I20240503 14:04:29.174146 5805 raft_consensus.cc:482] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.174441 5805 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.178915 5881 raft_consensus.cc:504] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.179759 5805 raft_consensus.cc:504] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.180696 5881 leader_election.cc:288] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:29.182475 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:29.183087 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:29.183148 5806 raft_consensus.cc:3032] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.183351 5805 leader_election.cc:288] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:29.183779 5806 raft_consensus.cc:729] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.183960 5881 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.184882 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7629315cc19b4c15a9f1e725df1dcb0b" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.184640 5806 consensus_queue.cc:260] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.185510 5955 raft_consensus.cc:3032] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.185889 5955 raft_consensus.cc:729] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:29.183957 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.186448 5954 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.186707 5955 consensus_queue.cc:260] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.187990 5955 raft_consensus.cc:3037] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.188303 5806 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.192296 5881 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.193845 5772 leader_election.cc:302] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:29.194818 6017 raft_consensus.cc:2781] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.195322 6017 raft_consensus.cc:686] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.196172 6017 consensus_queue.cc:237] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.198601 5954 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.200011 5847 leader_election.cc:302] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:29.200887 6014 raft_consensus.cc:2781] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.192152 6004 raft_consensus.cc:979] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: : Instructing follower 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:29.201296 6014 raft_consensus.cc:686] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:29.201611 6004 raft_consensus.cc:1067] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Signalling peer 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:29.202080 6014 consensus_queue.cc:237] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.203348 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:53390
I20240503 14:04:29.204126 5881 raft_consensus.cc:482] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.204595 5881 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.207036 5955 raft_consensus.cc:2454] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.209751 5700 catalog_manager.cc:5484] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 1 to 2, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 2 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.212534 5881 raft_consensus.cc:504] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.241952 5806 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.242234 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:29.242874 5805 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.246235 5881 leader_election.cc:288] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:29.247279 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.247866 5954 raft_consensus.cc:3032] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.248164 5954 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:29.248862 5954 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.249862 5954 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.251001 5805 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.252293 5846 leader_election.cc:302] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:29.252992 6014 raft_consensus.cc:2781] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.253468 6014 raft_consensus.cc:686] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:29.254302 6014 consensus_queue.cc:237] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.255894 5701 catalog_manager.cc:5484] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 1 to 2, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.256886 5954 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 2.
I20240503 14:04:29.259997 6027 raft_consensus.cc:979] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:29.260550 6017 raft_consensus.cc:1067] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Signalling peer 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:29.262084 5954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d"
dest_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:56332
I20240503 14:04:29.262610 5954 raft_consensus.cc:482] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.262985 5954 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.269881 5954 raft_consensus.cc:504] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.271553 5954 leader_election.cc:288] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 2 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:29.271328 5701 catalog_manager.cc:5484] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 1 to 2, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 2 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.272912 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:29.273555 5881 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.274775 5805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf18ff07174497a9554a9fcb513af3d" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:29.275434 5805 raft_consensus.cc:3032] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.275782 5805 raft_consensus.cc:729] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.276635 5805 consensus_queue.cc:260] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.277940 5805 raft_consensus.cc:3037] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.280926 5881 raft_consensus.cc:2454] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 2.
I20240503 14:04:29.282124 5921 leader_election.cc:302] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:29.283020 6037 raft_consensus.cc:2781] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.283475 6037 raft_consensus.cc:686] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:29.284363 6037 consensus_queue.cc:237] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.284807 5805 raft_consensus.cc:2454] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 2.
I20240503 14:04:29.291112 5701 catalog_manager.cc:5484] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 1 to 2, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 2 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:29.367240 6084 raft_consensus.cc:979] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.367687 6014 raft_consensus.cc:1067] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.369220 5805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41514
I20240503 14:04:29.369761 5805 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.370132 5805 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.375877 5805 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.377511 5805 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:29.378562 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:29.378736 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.379110 5881 raft_consensus.cc:3032] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.379247 5954 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.379413 5881 raft_consensus.cc:729] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:29.380214 5881 consensus_queue.cc:260] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.381373 5881 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.384841 5954 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.386081 5772 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:29.386677 5881 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.386826 6027 raft_consensus.cc:2781] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.387405 6027 raft_consensus.cc:686] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.388252 6027 consensus_queue.cc:237] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:29.394783 5701 catalog_manager.cc:5484] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 1 to 2, leader changed from 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 2 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:29.430270 6004 raft_consensus.cc:979] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.430830 6037 raft_consensus.cc:1067] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:29.432418 5805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c254be31c226480ba71d7231d79ee234"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41520
I20240503 14:04:29.432916 5805 raft_consensus.cc:482] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:29.433211 5805 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.438123 5805 raft_consensus.cc:504] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:29.440035 5805 leader_election.cc:288] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Requested vote from peers 913051779cda40fc82f6279197502e7b (127.4.250.195:45795), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:29.441001 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:29.441102 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c254be31c226480ba71d7231d79ee234" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:29.441581 5954 raft_consensus.cc:3032] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:29.441687 5881 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.441893 5954 raft_consensus.cc:729] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:29.442595 5954 consensus_queue.cc:260] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:29.443535 5954 raft_consensus.cc:3037] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:29.446655 5881 raft_consensus.cc:2454] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.447762 5772 leader_election.cc:302] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:29.448668 6027 raft_consensus.cc:2781] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:29.448714 5954 raft_consensus.cc:2454] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 2.
I20240503 14:04:29.449168 6027 raft_consensus.cc:686] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:29.450076 6027 consensus_queue.cc:237] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } }
I20240503 14:04:29.457710 5701 catalog_manager.cc:5484] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 1 to 2, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 2 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:29.551954 5881 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.553361 6027 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.560961 5954 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.562798 6017 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.594480 5954 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.595894 6014 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.604934 5805 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.606662 6014 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.660259 5954 raft_consensus.cc:1261] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.661518 6014 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.669525 5805 raft_consensus.cc:1261] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.671099 6002 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.672616 5954 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.673905 6084 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.686017 5806 raft_consensus.cc:1261] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.686007 5804 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.687995 6014 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.688374 6037 consensus_queue.cc:1035] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.701478 5881 raft_consensus.cc:1261] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.703631 6031 consensus_queue.cc:1035] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.706313 5954 raft_consensus.cc:1261] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.707398 5879 raft_consensus.cc:1261] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.707770 6028 consensus_queue.cc:1035] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.708982 6017 consensus_queue.cc:1035] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.817375 5879 raft_consensus.cc:1261] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.818604 6028 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.826925 5954 raft_consensus.cc:1261] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.828576 6028 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.853946 5879 raft_consensus.cc:1261] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.855119 6028 consensus_queue.cc:1035] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:29.863942 5954 raft_consensus.cc:1261] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:29.865595 6097 consensus_queue.cc:1035] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:31.968855 5099 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240503 14:04:31.969698 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:31.991957 5804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:31.992630 5804 raft_consensus.cc:595] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Received request to transfer leadership to TS 913051779cda40fc82f6279197502e7b
I20240503 14:04:31.995048 5804 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:31.995657 5804 raft_consensus.cc:595] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Received request to transfer leadership to TS 913051779cda40fc82f6279197502e7b
I20240503 14:04:31.996676 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:04:31.997128 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:32.029100 6103 raft_consensus.cc:979] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:32.029563 6103 raft_consensus.cc:1067] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Signalling peer 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:32.030969 5954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:56332
I20240503 14:04:32.031431 5954 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:32.031687 5954 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:32.035828 5954 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.037425 5954 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 3 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:32.038375 5804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:32.038622 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:32.039104 5804 raft_consensus.cc:3032] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:32.039155 5879 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:32.039577 5804 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:32.040436 5804 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.041785 5804 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:32.046453 5879 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 3.
I20240503 14:04:32.047827 5921 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:32.048485 5804 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 3.
I20240503 14:04:32.048767 6100 raft_consensus.cc:2781] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:32.049966 6100 raft_consensus.cc:686] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:32.050750 6100 consensus_queue.cc:237] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.057525 5700 catalog_manager.cc:5484] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 2 to 3, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 3 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:32.153539 6097 raft_consensus.cc:979] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:32.153995 6103 raft_consensus.cc:1067] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Signalling peer 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:32.155467 5954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:56332
I20240503 14:04:32.155925 5954 raft_consensus.cc:482] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:32.156215 5954 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:32.160611 5954 raft_consensus.cc:504] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.162037 5954 leader_election.cc:288] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 2 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:32.162951 5804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:32.163089 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:32.163519 5804 raft_consensus.cc:3032] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:32.163591 5879 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:32.163960 5804 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:32.164692 5804 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.165625 5804 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:32.168713 5879 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 2.
I20240503 14:04:32.169814 5921 leader_election.cc:302] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:32.170642 6100 raft_consensus.cc:2781] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:04:32.170722 5804 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 2.
I20240503 14:04:32.171133 6100 raft_consensus.cc:686] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:32.172073 6100 consensus_queue.cc:237] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:32.178992 5700 catalog_manager.cc:5484] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 1 to 2, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 2 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:32.543510 5879 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:32.544920 6100 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:32.561574 5804 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:32.565340 6100 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:32.676419 5879 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:32.676796 5804 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:04:32.677675 6113 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:32.678411 6100 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:04:32.997980 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:33.003422 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:33.003840 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:34.004647 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:34.010031 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:34.010462 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:35.011406 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:35.017818 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:35.018404 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:36.019331 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:36.025276 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:36.025846 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:37.026747 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:37.031888 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:37.032385 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:38.033202 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:38.038368 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:38.038838 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:39.039630 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:39.044785 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:39.045244 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:40.046111 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:40.051389 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:40.051868 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:41.052796 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:41.057987 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:41.058400 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:42.082772 5099 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240503 14:04:42.091666 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:42.092370 5955 raft_consensus.cc:595] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:42.099246 5955 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "913051779cda40fc82f6279197502e7b"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:56338
I20240503 14:04:42.099694 5955 raft_consensus.cc:595] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:42.103963 5879 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:53392
I20240503 14:04:42.104444 5879 raft_consensus.cc:595] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:42.108700 5879 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:53392
I20240503 14:04:42.109107 5879 raft_consensus.cc:595] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:42.113087 5879 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
mode: GRACEFUL
new_leader_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:53392
I20240503 14:04:42.113512 5879 raft_consensus.cc:595] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Received request to transfer leadership to TS 9f7a820b09714c96aec13be58ab289ab
I20240503 14:04:42.198870 6129 raft_consensus.cc:979] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.199412 6124 raft_consensus.cc:1067] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.201025 5806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41514
I20240503 14:04:42.201582 5806 raft_consensus.cc:482] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:42.201936 5806 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.207612 5806 raft_consensus.cc:504] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.209460 5806 leader_election.cc:288] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:42.210522 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:42.210299 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:42.211076 5955 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.211144 5879 raft_consensus.cc:3032] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:42.211519 5879 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:42.212335 5879 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.213850 6128 raft_consensus.cc:979] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.213747 5879 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.214299 6124 raft_consensus.cc:1067] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.215608 5806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41514
I20240503 14:04:42.216212 5806 raft_consensus.cc:482] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:42.216657 5806 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.217885 5955 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.219291 5772 leader_election.cc:302] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:42.220191 6120 raft_consensus.cc:2781] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:42.220679 5879 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.221266 6120 raft_consensus.cc:686] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:42.222275 6120 consensus_queue.cc:237] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.223050 5806 raft_consensus.cc:504] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.225611 5806 leader_election.cc:288] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:42.226639 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:42.227371 5955 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.230623 5704 catalog_manager.cc:5484] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 2 to 3, leader changed from 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 3 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:42.231982 5955 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.232550 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:42.233098 5772 leader_election.cc:302] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 913051779cda40fc82f6279197502e7b, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:42.233263 5879 raft_consensus.cc:3032] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:42.233641 5879 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:42.233842 6120 raft_consensus.cc:2781] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:42.234395 6120 raft_consensus.cc:686] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:42.234447 5879 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.235628 5879 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.235436 6120 consensus_queue.cc:237] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.242717 5879 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.254074 5704 catalog_manager.cc:5484] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 2 to 3, leader changed from 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 3 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:42.301074 6138 raft_consensus.cc:979] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.301620 6121 raft_consensus.cc:1067] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.302918 5806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41520
I20240503 14:04:42.303388 5806 raft_consensus.cc:482] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:42.303717 5806 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:42.308691 5806 raft_consensus.cc:504] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.310281 5806 leader_election.cc:288] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 4 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:42.311076 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:42.311228 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43220acdc744b6ac9c98e97bb91ba7" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:42.311753 5955 raft_consensus.cc:3032] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:42.311719 5881 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:42.312206 5955 raft_consensus.cc:729] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:42.312788 5955 consensus_queue.cc:260] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.313669 5955 raft_consensus.cc:3037] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:42.317353 5881 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 4.
I20240503 14:04:42.317667 5955 raft_consensus.cc:2454] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 4.
I20240503 14:04:42.318441 5772 leader_election.cc:302] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:42.319032 6120 raft_consensus.cc:2781] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:42.319398 6120 raft_consensus.cc:686] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 4 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:42.320179 6120 consensus_queue.cc:237] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.326056 5704 catalog_manager.cc:5484] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 3 to 4, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 4 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:42.350235 6138 raft_consensus.cc:979] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.350697 6121 raft_consensus.cc:1067] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.352035 5806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41520
I20240503 14:04:42.352555 5806 raft_consensus.cc:482] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:42.352830 5806 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.357336 5806 raft_consensus.cc:504] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.358825 5806 leader_election.cc:288] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:42.359721 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:42.359776 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:42.360388 5881 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.360445 5955 raft_consensus.cc:3032] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:42.360817 5955 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:42.361346 5955 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.362175 5955 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.364851 5881 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.365718 5772 leader_election.cc:302] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:42.366348 6120 raft_consensus.cc:2781] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:42.366299 5955 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.366921 6120 raft_consensus.cc:686] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:42.367664 6120 consensus_queue.cc:237] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.374064 5704 catalog_manager.cc:5484] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 2 to 3, leader changed from 913051779cda40fc82f6279197502e7b (127.4.250.195) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 3 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:42.682684 5881 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.683948 6120 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:42.691147 6128 raft_consensus.cc:979] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: : Instructing follower 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.691721 6124 raft_consensus.cc:1067] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Signalling peer 9f7a820b09714c96aec13be58ab289ab to start an election
I20240503 14:04:42.696424 5806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
from {username='slave'} at 127.0.0.1:41514
I20240503 14:04:42.696939 5955 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.697127 5806 raft_consensus.cc:482] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:42.697564 5806 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.698716 6130 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:42.704931 5881 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:42.707720 6130 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:04:42.711818 5806 raft_consensus.cc:504] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.714504 5806 leader_election.cc:288] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Requested vote from peers 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:42.715792 5879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:42.716380 5879 raft_consensus.cc:3032] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:42.716660 5879 raft_consensus.cc:729] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:42.717306 5879 consensus_queue.cc:260] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.718566 5879 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.717319 5955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e46ad74f5054c388807beb37a70fd8f" candidate_uuid: "9f7a820b09714c96aec13be58ab289ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:42.725389 5955 raft_consensus.cc:3037] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:42.729631 5879 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.730773 5772 leader_election.cc:302] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 9f7a820b09714c96aec13be58ab289ab; no voters:
I20240503 14:04:42.730962 5954 raft_consensus.cc:1261] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:42.731720 6133 raft_consensus.cc:2781] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:42.732317 6133 raft_consensus.cc:686] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Leader. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:42.733433 6133 consensus_queue.cc:237] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:42.736397 6120 consensus_queue.cc:1035] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:42.738682 5955 raft_consensus.cc:2454] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f7a820b09714c96aec13be58ab289ab in term 3.
I20240503 14:04:42.747582 5704 catalog_manager.cc:5484] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab reported cstate change: term changed from 2 to 3, leader changed from 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194) to 9f7a820b09714c96aec13be58ab289ab (127.4.250.193). New cstate: current_term: 3 leader_uuid: "9f7a820b09714c96aec13be58ab289ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:42.753688 5879 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.755916 6133 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:42.765298 5954 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.767796 6120 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:42.789120 5954 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.790205 6120 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:42.803104 5879 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:42.804208 6120 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:43.138309 5954 raft_consensus.cc:1261] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:43.139554 6120 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:43.149787 5879 raft_consensus.cc:1261] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 9f7a820b09714c96aec13be58ab289ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:43.152060 6120 consensus_queue.cc:1035] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:45.149849 5099 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240503 14:04:45.150879 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:45.158397 5806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:45.159096 5806 raft_consensus.cc:595] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Received request to transfer leadership to TS 913051779cda40fc82f6279197502e7b
I20240503 14:04:45.161676 5806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:45.162181 5806 raft_consensus.cc:595] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Received request to transfer leadership to TS 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:45.164665 5806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:45.165330 5806 raft_consensus.cc:595] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Received request to transfer leadership to TS 913051779cda40fc82f6279197502e7b
I20240503 14:04:45.167707 5806 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
mode: GRACEFUL
new_leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:48360
I20240503 14:04:45.168210 5806 raft_consensus.cc:595] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Received request to transfer leadership to TS 79ba0c2d1c934cf291e699f8e9e70400
I20240503 14:04:45.169188 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240503 14:04:45.169662 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:45.184484 6130 raft_consensus.cc:979] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:45.184990 6158 raft_consensus.cc:1067] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Signalling peer 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:45.186501 5954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638"
dest_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:56332
I20240503 14:04:45.186966 5954 raft_consensus.cc:482] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:45.187438 5954 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.193746 5954 raft_consensus.cc:504] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.195993 5954 leader_election.cc:288] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 4 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:45.197304 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:45.197157 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bce6196cffa46549a33cf717d7a3638" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:45.197942 5806 raft_consensus.cc:3032] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:45.198000 5881 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.198380 5806 raft_consensus.cc:729] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:45.199204 5806 consensus_queue.cc:260] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.200590 5806 raft_consensus.cc:3037] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.205495 5881 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 4.
I20240503 14:04:45.206876 5921 leader_election.cc:302] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:45.207537 5806 raft_consensus.cc:2454] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 4.
I20240503 14:04:45.207684 6121 raft_consensus.cc:2781] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:45.208837 6121 raft_consensus.cc:686] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 4 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:45.209730 6121 consensus_queue.cc:237] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.216502 5704 catalog_manager.cc:5484] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 3 to 4, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 4 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:45.352707 6130 raft_consensus.cc:979] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:45.353197 6158 raft_consensus.cc:1067] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Signalling peer 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:45.354766 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:53384
I20240503 14:04:45.355248 5881 raft_consensus.cc:482] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:45.355509 5881 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.360149 5881 raft_consensus.cc:504] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.361994 5881 leader_election.cc:288] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 4 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:45.362953 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:45.363101 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3ec7afab9254f5fa4db7ef9bd18aec9" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:45.363548 5806 raft_consensus.cc:3032] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:45.363600 5954 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.363834 5806 raft_consensus.cc:729] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:45.364562 5806 consensus_queue.cc:260] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.365600 5806 raft_consensus.cc:3037] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.368762 5954 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 4.
I20240503 14:04:45.370127 5847 leader_election.cc:302] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:45.371045 5806 raft_consensus.cc:2454] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 4.
I20240503 14:04:45.371088 6124 raft_consensus.cc:2781] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:45.372509 6124 raft_consensus.cc:686] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:45.373487 6124 consensus_queue.cc:237] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.380651 5704 catalog_manager.cc:5484] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 3 to 4, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 4 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:45.554147 6130 raft_consensus.cc:979] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:45.554618 6158 raft_consensus.cc:1067] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Signalling peer 79ba0c2d1c934cf291e699f8e9e70400 to start an election
I20240503 14:04:45.556113 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483"
dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
from {username='slave'} at 127.0.0.1:53384
I20240503 14:04:45.556560 5881 raft_consensus.cc:482] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:45.556813 5881 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.561163 5881 raft_consensus.cc:504] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.562623 5881 leader_election.cc:288] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 4 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 913051779cda40fc82f6279197502e7b (127.4.250.195:45795)
I20240503 14:04:45.563598 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:45.563628 5954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762e19a422db4495a8d7cd33ccb7e483" candidate_uuid: "79ba0c2d1c934cf291e699f8e9e70400" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "913051779cda40fc82f6279197502e7b"
I20240503 14:04:45.564247 5806 raft_consensus.cc:3032] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:04:45.564275 5954 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.564584 5806 raft_consensus.cc:729] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:45.565196 5806 consensus_queue.cc:260] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.566146 5806 raft_consensus.cc:3037] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:04:45.569169 5954 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 4.
I20240503 14:04:45.570174 5847 leader_election.cc:302] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:45.570993 6124 raft_consensus.cc:2781] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:04:45.570994 5806 raft_consensus.cc:2454] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ba0c2d1c934cf291e699f8e9e70400 in term 4.
I20240503 14:04:45.571456 6124 raft_consensus.cc:686] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 LEADER]: Becoming Leader. State: Replica: 79ba0c2d1c934cf291e699f8e9e70400, State: Running, Role: LEADER
I20240503 14:04:45.572451 6124 consensus_queue.cc:237] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.579916 5704 catalog_manager.cc:5484] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 reported cstate change: term changed from 3 to 4, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194). New cstate: current_term: 4 leader_uuid: "79ba0c2d1c934cf291e699f8e9e70400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:45.681448 5881 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:45.682633 6121 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:45.692813 5806 raft_consensus.cc:1261] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:45.694787 6159 consensus_queue.cc:1035] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:04:45.747532 6130 raft_consensus.cc:979] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: : Instructing follower 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:45.748111 6130 raft_consensus.cc:1067] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Signalling peer 913051779cda40fc82f6279197502e7b to start an election
I20240503 14:04:45.749615 5954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3"
dest_uuid: "913051779cda40fc82f6279197502e7b"
from {username='slave'} at 127.0.0.1:56332
I20240503 14:04:45.750159 5954 raft_consensus.cc:482] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:04:45.750526 5954 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:45.755646 5954 raft_consensus.cc:504] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.757042 5954 leader_election.cc:288] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 3 election: Requested vote from peers 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397), 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853)
I20240503 14:04:45.757846 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f7a820b09714c96aec13be58ab289ab"
I20240503 14:04:45.758086 5881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b05f61b04c4c948c9229d79112f8d3" candidate_uuid: "913051779cda40fc82f6279197502e7b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "79ba0c2d1c934cf291e699f8e9e70400"
I20240503 14:04:45.758371 5806 raft_consensus.cc:3032] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:04:45.758592 5881 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:45.758651 5806 raft_consensus.cc:729] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f7a820b09714c96aec13be58ab289ab, State: Running, Role: LEADER
I20240503 14:04:45.759382 5806 consensus_queue.cc:260] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.760331 5806 raft_consensus.cc:3037] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:04:45.763118 5881 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 3.
I20240503 14:04:45.764070 5921 leader_election.cc:302] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ba0c2d1c934cf291e699f8e9e70400, 913051779cda40fc82f6279197502e7b; no voters:
I20240503 14:04:45.764918 6159 raft_consensus.cc:2781] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:04:45.765069 5806 raft_consensus.cc:2454] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913051779cda40fc82f6279197502e7b in term 3.
I20240503 14:04:45.765291 6159 raft_consensus.cc:686] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Becoming Leader. State: Replica: 913051779cda40fc82f6279197502e7b, State: Running, Role: LEADER
I20240503 14:04:45.766155 6159 consensus_queue.cc:237] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } }
I20240503 14:04:45.773304 5701 catalog_manager.cc:5484] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b reported cstate change: term changed from 2 to 3, leader changed from 9f7a820b09714c96aec13be58ab289ab (127.4.250.193) to 913051779cda40fc82f6279197502e7b (127.4.250.195). New cstate: current_term: 3 leader_uuid: "913051779cda40fc82f6279197502e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:45.840391 5806 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:45.841557 6124 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:45.850908 5954 raft_consensus.cc:1261] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:45.852788 6176 consensus_queue.cc:1035] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:04:46.135412 5954 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:46.136579 6176 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "913051779cda40fc82f6279197502e7b" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:46.145398 5806 raft_consensus.cc:1261] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Refusing update from remote peer 79ba0c2d1c934cf291e699f8e9e70400: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240503 14:04:46.147019 6176 consensus_queue.cc:1035] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:04:46.170423 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:46.175868 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:46.176334 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:46.185617 5806 raft_consensus.cc:1261] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:46.186872 6178 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f7a820b09714c96aec13be58ab289ab" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 43397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:04:46.195745 5881 raft_consensus.cc:1261] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Refusing update from remote peer 913051779cda40fc82f6279197502e7b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240503 14:04:46.198139 6178 consensus_queue.cc:1035] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ba0c2d1c934cf291e699f8e9e70400" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 46853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:04:47.177141 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:47.182518 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:47.182950 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:48.183820 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:48.189282 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:48.189749 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:49.190575 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:49.196610 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:49.197055 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:50.197840 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:50.203207 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:50.203656 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:51.204538 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:51.209748 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:51.210217 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:52.210968 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:52.216338 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:52.216773 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:53.217593 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:53.222903 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:53.223343 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:54.224267 5099 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:04:54.229784 5099 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:04:54.230237 5099 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:04:55.258450 5099 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240503 14:04:55.259048 5099 tablet_server.cc:178] TabletServer@127.4.250.193:0 shutting down...
W20240503 14:04:55.271030 5846 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20240503 14:04:55.275111 5846 consensus_peers.cc:485] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 -> Peer 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397): Couldn't send request to peer 9f7a820b09714c96aec13be58ab289ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:55.286778 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:55.287576 5099 tablet_replica.cc:330] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.288615 5099 raft_consensus.cc:2229] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.289443 5099 raft_consensus.cc:2258] T 93fd44cd62ff467fa8b2cc596459e721 P 9f7a820b09714c96aec13be58ab289ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.291373 5099 tablet_replica.cc:330] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.291891 5099 raft_consensus.cc:2229] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.292757 5099 raft_consensus.cc:2258] T 7629315cc19b4c15a9f1e725df1dcb0b P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.294651 5099 tablet_replica.cc:330] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.295228 5099 raft_consensus.cc:2229] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.295724 5099 raft_consensus.cc:2258] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.297578 5099 tablet_replica.cc:330] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.298041 5099 raft_consensus.cc:2229] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.298424 5099 raft_consensus.cc:2258] T 762e19a422db4495a8d7cd33ccb7e483 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.300177 5099 tablet_replica.cc:330] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.300626 5099 raft_consensus.cc:2229] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.301347 5099 raft_consensus.cc:2258] T c254be31c226480ba71d7231d79ee234 P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.303063 5099 tablet_replica.cc:330] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.303540 5099 raft_consensus.cc:2229] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.303931 5099 raft_consensus.cc:2258] T 47b05f61b04c4c948c9229d79112f8d3 P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.305720 5099 tablet_replica.cc:330] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.306145 5099 raft_consensus.cc:2229] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.306524 5099 raft_consensus.cc:2258] T 8bf18ff07174497a9554a9fcb513af3d P 9f7a820b09714c96aec13be58ab289ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.308063 5099 tablet_replica.cc:330] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.308566 5099 raft_consensus.cc:2229] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.308935 5099 raft_consensus.cc:2258] T 7bce6196cffa46549a33cf717d7a3638 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.310550 5099 tablet_replica.cc:330] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.310992 5099 raft_consensus.cc:2229] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.311805 5099 raft_consensus.cc:2258] T ed43220acdc744b6ac9c98e97bb91ba7 P 9f7a820b09714c96aec13be58ab289ab [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.313490 5099 tablet_replica.cc:330] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab: stopping tablet replica
I20240503 14:04:55.313907 5099 raft_consensus.cc:2229] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.314736 5099 raft_consensus.cc:2258] T 2e46ad74f5054c388807beb37a70fd8f P 9f7a820b09714c96aec13be58ab289ab [term 3 FOLLOWER]: Raft consensus is shut down!
W20240503 14:04:55.332266 5772 connection.cc:441] server connection from 127.0.0.1:41520 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41520 (request call id 171) could send its response
W20240503 14:04:55.333370 5920 consensus_peers.cc:485] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b -> Peer 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397): Couldn't send request to peer 9f7a820b09714c96aec13be58ab289ab. Status: Network error: recv got EOF from 127.4.250.193:43397 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:55.346975 5099 tablet_server.cc:194] TabletServer@127.4.250.193:0 shutdown complete.
I20240503 14:04:55.365681 5099 tablet_server.cc:178] TabletServer@127.4.250.194:0 shutting down...
W20240503 14:04:55.372279 5921 consensus_peers.cc:485] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b -> Peer 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853): Couldn't send request to peer 79ba0c2d1c934cf291e699f8e9e70400. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:04:55.389429 5921 consensus_peers.cc:485] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b -> Peer 79ba0c2d1c934cf291e699f8e9e70400 (127.4.250.194:46853): Couldn't send request to peer 79ba0c2d1c934cf291e699f8e9e70400. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:55.397161 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:55.397940 5099 tablet_replica.cc:330] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.398984 5099 raft_consensus.cc:2229] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.399763 5099 raft_consensus.cc:2258] T c254be31c226480ba71d7231d79ee234 P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.402383 5099 tablet_replica.cc:330] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.402967 5099 raft_consensus.cc:2229] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240503 14:04:55.403463 5920 consensus_peers.cc:485] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b -> Peer 9f7a820b09714c96aec13be58ab289ab (127.4.250.193:43397): Couldn't send request to peer 9f7a820b09714c96aec13be58ab289ab. Status: Network error: Client connection negotiation failed: client connection to 127.4.250.193:43397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:55.403467 5099 raft_consensus.cc:2258] T 7629315cc19b4c15a9f1e725df1dcb0b P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.405632 5099 tablet_replica.cc:330] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.406180 5099 raft_consensus.cc:2229] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.407034 5099 raft_consensus.cc:2258] T 93fd44cd62ff467fa8b2cc596459e721 P 79ba0c2d1c934cf291e699f8e9e70400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.408843 5099 tablet_replica.cc:330] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.409365 5099 raft_consensus.cc:2229] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.410236 5099 raft_consensus.cc:2258] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.412235 5099 tablet_replica.cc:330] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.412735 5099 raft_consensus.cc:2229] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.413584 5099 raft_consensus.cc:2258] T 762e19a422db4495a8d7cd33ccb7e483 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.415421 5099 tablet_replica.cc:330] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.415953 5099 raft_consensus.cc:2229] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.416450 5099 raft_consensus.cc:2258] T 47b05f61b04c4c948c9229d79112f8d3 P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.418115 5099 tablet_replica.cc:330] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.418586 5099 raft_consensus.cc:2229] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.418987 5099 raft_consensus.cc:2258] T 8bf18ff07174497a9554a9fcb513af3d P 79ba0c2d1c934cf291e699f8e9e70400 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.420766 5099 tablet_replica.cc:330] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.421238 5099 raft_consensus.cc:2229] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.421734 5099 raft_consensus.cc:2258] T ed43220acdc744b6ac9c98e97bb91ba7 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.423545 5099 tablet_replica.cc:330] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.424014 5099 raft_consensus.cc:2229] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.424542 5099 raft_consensus.cc:2258] T 7bce6196cffa46549a33cf717d7a3638 P 79ba0c2d1c934cf291e699f8e9e70400 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.426157 5099 tablet_replica.cc:330] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400: stopping tablet replica
I20240503 14:04:55.426596 5099 raft_consensus.cc:2229] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.426990 5099 raft_consensus.cc:2258] T 2e46ad74f5054c388807beb37a70fd8f P 79ba0c2d1c934cf291e699f8e9e70400 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.455242 5099 tablet_server.cc:194] TabletServer@127.4.250.194:0 shutdown complete.
I20240503 14:04:55.475085 5099 tablet_server.cc:178] TabletServer@127.4.250.195:0 shutting down...
I20240503 14:04:55.498790 5099 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:04:55.499441 5099 tablet_replica.cc:330] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.500015 5099 raft_consensus.cc:2229] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 4 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.500836 5099 raft_consensus.cc:2258] T 7bce6196cffa46549a33cf717d7a3638 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.502743 5099 tablet_replica.cc:330] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.503260 5099 raft_consensus.cc:2229] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.503719 5099 raft_consensus.cc:2258] T ed43220acdc744b6ac9c98e97bb91ba7 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.505800 5099 tablet_replica.cc:330] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.506429 5099 raft_consensus.cc:2229] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 3 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.507292 5099 raft_consensus.cc:2258] T 47b05f61b04c4c948c9229d79112f8d3 P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.509194 5099 tablet_replica.cc:330] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.509742 5099 raft_consensus.cc:2229] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 2 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.510617 5099 raft_consensus.cc:2258] T 8bf18ff07174497a9554a9fcb513af3d P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.512445 5099 tablet_replica.cc:330] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.512949 5099 raft_consensus.cc:2229] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.513388 5099 raft_consensus.cc:2258] T 2e46ad74f5054c388807beb37a70fd8f P 913051779cda40fc82f6279197502e7b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.515018 5099 tablet_replica.cc:330] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.515477 5099 raft_consensus.cc:2229] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.515892 5099 raft_consensus.cc:2258] T 7629315cc19b4c15a9f1e725df1dcb0b P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.517575 5099 tablet_replica.cc:330] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.518007 5099 raft_consensus.cc:2229] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.518378 5099 raft_consensus.cc:2258] T c254be31c226480ba71d7231d79ee234 P 913051779cda40fc82f6279197502e7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.519891 5099 tablet_replica.cc:330] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.520361 5099 raft_consensus.cc:2229] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.520714 5099 raft_consensus.cc:2258] T 93fd44cd62ff467fa8b2cc596459e721 P 913051779cda40fc82f6279197502e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.522205 5099 tablet_replica.cc:330] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.522635 5099 raft_consensus.cc:2229] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.523001 5099 raft_consensus.cc:2258] T a3ec7afab9254f5fa4db7ef9bd18aec9 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.524636 5099 tablet_replica.cc:330] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b: stopping tablet replica
I20240503 14:04:55.525031 5099 raft_consensus.cc:2229] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:04:55.525413 5099 raft_consensus.cc:2258] T 762e19a422db4495a8d7cd33ccb7e483 P 913051779cda40fc82f6279197502e7b [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.553498 5099 tablet_server.cc:194] TabletServer@127.4.250.195:0 shutdown complete.
I20240503 14:04:55.571249 5099 master.cc:537] Master@127.4.250.254:46567 shutting down...
I20240503 14:04:55.592819 5099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:04:55.593423 5099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:04:55.593773 5099 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc7ff289dd6e460c986a1f75cac4e208: stopping tablet replica
I20240503 14:04:55.612285 5099 master.cc:552] Master@127.4.250.254:46567 shutdown complete.
79ba0c2d1c934cf291e699f8e9e70400 4
913051779cda40fc82f6279197502e7b 1
9f7a820b09714c96aec13be58ab289ab 5
79ba0c2d1c934cf291e699f8e9e70400 4
913051779cda40fc82f6279197502e7b 3
9f7a820b09714c96aec13be58ab289ab 3
79ba0c2d1c934cf291e699f8e9e70400 1
913051779cda40fc82f6279197502e7b 1
9f7a820b09714c96aec13be58ab289ab 8
79ba0c2d1c934cf291e699f8e9e70400 3
913051779cda40fc82f6279197502e7b 3
9f7a820b09714c96aec13be58ab289ab 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28581 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240503 14:04:55.650874 5099 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.250.254:36245
I20240503 14:04:55.651937 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:55.657662 6197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.657680 6196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.658756 6199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:55.659585 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:55.660529 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:55.660717 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:55.660840 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745095660829 us; error 0 us; skew 500 ppm
I20240503 14:04:55.661298 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:55.669431 5099 webserver.cc:460] Webserver started at http://127.4.250.254:45925/ using document root <none> and password file <none>
I20240503 14:04:55.669911 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:55.670069 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:55.670305 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:55.671459 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/master-0-root/instance:
uuid: "3497994a98a84200b12ab88b4ba50966"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:55.676998 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:04:55.680287 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:55.681020 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:55.681320 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/master-0-root
uuid: "3497994a98a84200b12ab88b4ba50966"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:55.681617 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:55.694646 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:55.695906 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:55.733757 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.254:36245
I20240503 14:04:55.733851 6255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.254:36245 every 8 connection(s)
I20240503 14:04:55.737711 6256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:55.747570 6256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966: Bootstrap starting.
I20240503 14:04:55.751837 6256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:55.756008 6256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966: No bootstrap required, opened a new log
I20240503 14:04:55.758045 6256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER }
I20240503 14:04:55.758535 6256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:55.758767 6256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3497994a98a84200b12ab88b4ba50966, State: Initialized, Role: FOLLOWER
I20240503 14:04:55.759348 6256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER }
I20240503 14:04:55.759816 6256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:04:55.760043 6256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:04:55.760322 6256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:55.764422 6256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER }
I20240503 14:04:55.764945 6256 leader_election.cc:302] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3497994a98a84200b12ab88b4ba50966; no voters:
I20240503 14:04:55.766103 6256 leader_election.cc:288] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:04:55.766489 6259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:55.767938 6259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [term 1 LEADER]: Becoming Leader. State: Replica: 3497994a98a84200b12ab88b4ba50966, State: Running, Role: LEADER
I20240503 14:04:55.768615 6259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER }
I20240503 14:04:55.769075 6256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:04:55.771600 6261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3497994a98a84200b12ab88b4ba50966. Latest consensus state: current_term: 1 leader_uuid: "3497994a98a84200b12ab88b4ba50966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER } }
I20240503 14:04:55.771587 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3497994a98a84200b12ab88b4ba50966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3497994a98a84200b12ab88b4ba50966" member_type: VOTER } }
I20240503 14:04:55.772354 6261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:55.772557 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966 [sys.catalog]: This master's current role is: LEADER
I20240503 14:04:55.780076 6264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:04:55.787258 6264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:04:55.788265 5099 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:04:55.795778 6264 catalog_manager.cc:1347] Generated new cluster ID: c6f5bcad04a34454ae9c4f56b7bb02b4
I20240503 14:04:55.796141 6264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:04:55.816000 6264 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:04:55.817309 6264 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:04:55.842979 6264 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3497994a98a84200b12ab88b4ba50966: Generated new TSK 0
I20240503 14:04:55.843644 6264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:04:55.855038 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:55.860759 6277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.862130 6278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.864452 6280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:55.864574 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:55.865562 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:55.865769 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:55.865932 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745095865915 us; error 0 us; skew 500 ppm
I20240503 14:04:55.866478 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:55.868892 5099 webserver.cc:460] Webserver started at http://127.4.250.193:35875/ using document root <none> and password file <none>
I20240503 14:04:55.869402 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:55.869587 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:55.869864 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:55.871042 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/instance:
uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:55.875890 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:55.879572 6285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:55.880437 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:55.880756 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root
uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:55.881081 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:55.925000 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:55.926226 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:55.927680 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:55.930023 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:55.930235 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:55.930480 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:55.930649 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:55.970252 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.193:37155
I20240503 14:04:55.970355 6347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.193:37155 every 8 connection(s)
I20240503 14:04:55.974750 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:55.981990 6352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.984107 6353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:55.988062 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:55.989903 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:55.989969 6348 heartbeater.cc:347] Connected to a master server at 127.4.250.254:36245
I20240503 14:04:55.990434 6348 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:55.991169 6348 heartbeater.cc:511] Master 127.4.250.254:36245 requested a full tablet report, sending...
I20240503 14:04:55.991223 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:55.991477 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:55.991633 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745095991616 us; error 0 us; skew 500 ppm
I20240503 14:04:55.992367 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:55.993283 6221 ts_manager.cc:194] Registered new tserver with Master: 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155)
I20240503 14:04:55.995137 5099 webserver.cc:460] Webserver started at http://127.4.250.194:34173/ using document root <none> and password file <none>
I20240503 14:04:55.995831 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:55.995795 6221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40806
I20240503 14:04:55.996254 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:55.996624 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:55.997959 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/instance:
uuid: "90a66da3b59e4511bdc1d35930fe558d"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:56.002385 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:04:56.005604 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.006359 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:04:56.006644 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root
uuid: "90a66da3b59e4511bdc1d35930fe558d"
format_stamp: "Formatted at 2024-05-03 14:04:55 on dist-test-slave-8wrt"
I20240503 14:04:56.006937 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:56.027648 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:56.028944 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:56.030351 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:56.032749 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:56.032966 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.033193 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:56.033346 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.071586 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.194:39097
I20240503 14:04:56.071681 6422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.194:39097 every 8 connection(s)
I20240503 14:04:56.076077 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:04:56.082661 6426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:56.083102 6427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:04:56.086341 6429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:04:56.086719 5099 server_base.cc:1034] running on GCE node
I20240503 14:04:56.087512 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:04:56.087735 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:04:56.087894 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745096087879 us; error 0 us; skew 500 ppm
I20240503 14:04:56.088598 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:04:56.088830 6423 heartbeater.cc:347] Connected to a master server at 127.4.250.254:36245
I20240503 14:04:56.089215 6423 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:56.090116 6423 heartbeater.cc:511] Master 127.4.250.254:36245 requested a full tablet report, sending...
I20240503 14:04:56.091300 5099 webserver.cc:460] Webserver started at http://127.4.250.195:37321/ using document root <none> and password file <none>
I20240503 14:04:56.091933 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:04:56.092211 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:04:56.092312 6221 ts_manager.cc:194] Registered new tserver with Master: 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:56.092563 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:04:56.094157 6221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40820
I20240503 14:04:56.094146 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/instance:
uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
format_stamp: "Formatted at 2024-05-03 14:04:56 on dist-test-slave-8wrt"
I20240503 14:04:56.100298 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:04:56.103531 6434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.104326 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:04:56.104626 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root
uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
format_stamp: "Formatted at 2024-05-03 14:04:56 on dist-test-slave-8wrt"
I20240503 14:04:56.104919 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:04:56.121405 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:04:56.122589 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:04:56.124049 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:04:56.126358 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:04:56.126572 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.126797 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:04:56.126964 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:04:56.165422 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.195:35583
I20240503 14:04:56.165521 6496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.195:35583 every 8 connection(s)
I20240503 14:04:56.178551 6497 heartbeater.cc:347] Connected to a master server at 127.4.250.254:36245
I20240503 14:04:56.179001 6497 heartbeater.cc:464] Registering TS with master...
I20240503 14:04:56.179989 6497 heartbeater.cc:511] Master 127.4.250.254:36245 requested a full tablet report, sending...
I20240503 14:04:56.181893 6221 ts_manager.cc:194] Registered new tserver with Master: 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:56.182114 5099 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013664361s
I20240503 14:04:56.183367 6221 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40828
I20240503 14:04:56.184494 5099 test_util.cc:268] Using random seed: 878092946
I20240503 14:04:56.207528 6221 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40842:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:04:56.213759 6221 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:04:56.399179 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 861ddaea971348a596840c74fb1e8a71 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 14:04:56.399951 6309 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd965edad81421d8b03f93895f80ebe (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 14:04:56.400154 6313 tablet_service.cc:1467] Processing CreateTablet for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 14:04:56.400689 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ddaea971348a596840c74fb1e8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.401300 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd965edad81421d8b03f93895f80ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.401968 6311 tablet_service.cc:1467] Processing CreateTablet for tablet d070812c38ea4aeda6ce94fac9ed7de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 14:04:56.403297 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d070812c38ea4aeda6ce94fac9ed7de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.410176 6310 tablet_service.cc:1467] Processing CreateTablet for tablet e388b249118b41faa4cb6efffb8fecea (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 14:04:56.411581 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388b249118b41faa4cb6efffb8fecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.411784 6462 tablet_service.cc:1467] Processing CreateTablet for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 14:04:56.412132 6461 tablet_service.cc:1467] Processing CreateTablet for tablet 861ddaea971348a596840c74fb1e8a71 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 14:04:56.412950 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.413117 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.413717 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ddaea971348a596840c74fb1e8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.418308 6460 tablet_service.cc:1467] Processing CreateTablet for tablet d070812c38ea4aeda6ce94fac9ed7de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 14:04:56.419175 6308 tablet_service.cc:1467] Processing CreateTablet for tablet 801af7dc07784c89890730a686553023 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 14:04:56.419749 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d070812c38ea4aeda6ce94fac9ed7de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.420954 6388 tablet_service.cc:1467] Processing CreateTablet for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240503 14:04:56.422232 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102ad45eeee4cb4a3d5b7f5656a70d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.425268 6387 tablet_service.cc:1467] Processing CreateTablet for tablet 861ddaea971348a596840c74fb1e8a71 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240503 14:04:56.426663 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ddaea971348a596840c74fb1e8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.429306 6386 tablet_service.cc:1467] Processing CreateTablet for tablet d070812c38ea4aeda6ce94fac9ed7de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240503 14:04:56.430671 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d070812c38ea4aeda6ce94fac9ed7de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.433246 6385 tablet_service.cc:1467] Processing CreateTablet for tablet e388b249118b41faa4cb6efffb8fecea (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 14:04:56.433696 6459 tablet_service.cc:1467] Processing CreateTablet for tablet e388b249118b41faa4cb6efffb8fecea (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240503 14:04:56.434564 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388b249118b41faa4cb6efffb8fecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.434962 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388b249118b41faa4cb6efffb8fecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.437182 6384 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd965edad81421d8b03f93895f80ebe (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 14:04:56.438560 6384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd965edad81421d8b03f93895f80ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.443974 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 6845d794633f4f47a07ae27a88362ffa (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 14:04:56.445473 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6845d794633f4f47a07ae27a88362ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.445710 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd965edad81421d8b03f93895f80ebe (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240503 14:04:56.446461 6456 tablet_service.cc:1467] Processing CreateTablet for tablet 6845d794633f4f47a07ae27a88362ffa (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 14:04:56.447016 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd965edad81421d8b03f93895f80ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.447479 6453 tablet_service.cc:1467] Processing CreateTablet for tablet 4f379675884943d6b074d7af75fdbf64 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 14:04:56.447865 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6845d794633f4f47a07ae27a88362ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.448812 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f379675884943d6b074d7af75fdbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.449196 6457 tablet_service.cc:1467] Processing CreateTablet for tablet 801af7dc07784c89890730a686553023 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 14:04:56.450466 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801af7dc07784c89890730a686553023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.454537 6455 tablet_service.cc:1467] Processing CreateTablet for tablet cbc690ef263249609cf947bfcd0d4170 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 14:04:56.454952 6305 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17c5d0f3d64427a915fdfd46f9d91f (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 14:04:56.455921 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc690ef263249609cf947bfcd0d4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.456175 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17c5d0f3d64427a915fdfd46f9d91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.458683 6304 tablet_service.cc:1467] Processing CreateTablet for tablet 4f379675884943d6b074d7af75fdbf64 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 14:04:56.453228 6306 tablet_service.cc:1467] Processing CreateTablet for tablet cbc690ef263249609cf947bfcd0d4170 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 14:04:56.460700 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17c5d0f3d64427a915fdfd46f9d91f (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 14:04:56.463225 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17c5d0f3d64427a915fdfd46f9d91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.469786 6383 tablet_service.cc:1467] Processing CreateTablet for tablet 801af7dc07784c89890730a686553023 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240503 14:04:56.472611 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801af7dc07784c89890730a686553023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.477131 6456 tablet_service.cc:1467] Processing CreateTablet for tablet ca47ba300cb146968aba2dc3e5f4857a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 14:04:56.477221 6381 tablet_service.cc:1467] Processing CreateTablet for tablet cbc690ef263249609cf947bfcd0d4170 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240503 14:04:56.477603 6382 tablet_service.cc:1467] Processing CreateTablet for tablet 6845d794633f4f47a07ae27a88362ffa (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240503 14:04:56.478699 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc690ef263249609cf947bfcd0d4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.479398 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6845d794633f4f47a07ae27a88362ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.460067 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f379675884943d6b074d7af75fdbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.484001 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc690ef263249609cf947bfcd0d4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.485203 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801af7dc07784c89890730a686553023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.485538 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca47ba300cb146968aba2dc3e5f4857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.489768 6380 tablet_service.cc:1467] Processing CreateTablet for tablet 3c17c5d0f3d64427a915fdfd46f9d91f (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240503 14:04:56.491923 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c17c5d0f3d64427a915fdfd46f9d91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.508288 6460 tablet_service.cc:1467] Processing CreateTablet for tablet ef212857f4874110a036926374b4d0cd (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 14:04:56.508760 6379 tablet_service.cc:1467] Processing CreateTablet for tablet 4f379675884943d6b074d7af75fdbf64 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240503 14:04:56.510092 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f379675884943d6b074d7af75fdbf64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.510078 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef212857f4874110a036926374b4d0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.527693 6311 tablet_service.cc:1467] Processing CreateTablet for tablet ca47ba300cb146968aba2dc3e5f4857a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 14:04:56.528298 6457 tablet_service.cc:1467] Processing CreateTablet for tablet ce2b1e76121d4515956175ba6bbe8fdb (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 14:04:56.529152 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca47ba300cb146968aba2dc3e5f4857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.540586 6309 tablet_service.cc:1467] Processing CreateTablet for tablet ef212857f4874110a036926374b4d0cd (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 14:04:56.558851 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef212857f4874110a036926374b4d0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.552070 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2b1e76121d4515956175ba6bbe8fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.544693 6461 tablet_service.cc:1467] Processing CreateTablet for tablet 7257be676307482195052c59227129f0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 14:04:56.566632 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7257be676307482195052c59227129f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.559441 6312 tablet_service.cc:1467] Processing CreateTablet for tablet ce2b1e76121d4515956175ba6bbe8fdb (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 14:04:56.572691 6383 tablet_service.cc:1467] Processing CreateTablet for tablet ca47ba300cb146968aba2dc3e5f4857a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240503 14:04:56.575750 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2b1e76121d4515956175ba6bbe8fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.594292 6459 tablet_service.cc:1467] Processing CreateTablet for tablet 56476ec0a3874a83bc22ec0c101088c8 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 14:04:56.597739 6305 tablet_service.cc:1467] Processing CreateTablet for tablet 56476ec0a3874a83bc22ec0c101088c8 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 14:04:56.599222 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56476ec0a3874a83bc22ec0c101088c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.599759 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca47ba300cb146968aba2dc3e5f4857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.609165 6313 tablet_service.cc:1467] Processing CreateTablet for tablet 6985cc0483c34fcd95b3e56b8b59bfb4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 14:04:56.610845 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6985cc0483c34fcd95b3e56b8b59bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.591578 6379 tablet_service.cc:1467] Processing CreateTablet for tablet ce2b1e76121d4515956175ba6bbe8fdb (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240503 14:04:56.616611 6306 tablet_service.cc:1467] Processing CreateTablet for tablet c4ee8e66a5b74a4394a4e339a0af957d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 14:04:56.591825 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0fcc21be834ead9ab016a33b25928c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 14:04:56.618067 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce2b1e76121d4515956175ba6bbe8fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.619051 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 7257be676307482195052c59227129f0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 14:04:56.618067 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ee8e66a5b74a4394a4e339a0af957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.624972 6454 tablet_service.cc:1467] Processing CreateTablet for tablet c4ee8e66a5b74a4394a4e339a0af957d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 14:04:56.633638 6312 tablet_service.cc:1467] Processing CreateTablet for tablet a4080dad66944ea291f6d69736cd2c23 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 14:04:56.595930 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56476ec0a3874a83bc22ec0c101088c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.645635 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0fcc21be834ead9ab016a33b25928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.648814 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0fcc21be834ead9ab016a33b25928c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 14:04:56.650938 6514 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:56.652489 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ee8e66a5b74a4394a4e339a0af957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.652730 6384 tablet_service.cc:1467] Processing CreateTablet for tablet ef212857f4874110a036926374b4d0cd (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240503 14:04:56.654042 6384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef212857f4874110a036926374b4d0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.657788 6455 tablet_service.cc:1467] Processing CreateTablet for tablet 6985cc0483c34fcd95b3e56b8b59bfb4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 14:04:56.659204 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6985cc0483c34fcd95b3e56b8b59bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.661967 6385 tablet_service.cc:1467] Processing CreateTablet for tablet 7257be676307482195052c59227129f0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240503 14:04:56.663334 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7257be676307482195052c59227129f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.635228 6310 tablet_service.cc:1467] Processing CreateTablet for tablet 66d5e8c5c12a43479ecc3382c9576c60 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 14:04:56.664660 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7257be676307482195052c59227129f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.668332 6512 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:56.668522 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0fcc21be834ead9ab016a33b25928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.674801 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4080dad66944ea291f6d69736cd2c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.675495 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d5e8c5c12a43479ecc3382c9576c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.667934 6381 tablet_service.cc:1467] Processing CreateTablet for tablet 6985cc0483c34fcd95b3e56b8b59bfb4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240503 14:04:56.681304 6513 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:56.685340 6462 tablet_service.cc:1467] Processing CreateTablet for tablet a4080dad66944ea291f6d69736cd2c23 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 14:04:56.687280 6514 tablet_bootstrap.cc:654] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:56.688727 6461 tablet_service.cc:1467] Processing CreateTablet for tablet 66d5e8c5c12a43479ecc3382c9576c60 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 14:04:56.691921 6380 tablet_service.cc:1467] Processing CreateTablet for tablet 56476ec0a3874a83bc22ec0c101088c8 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240503 14:04:56.693310 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56476ec0a3874a83bc22ec0c101088c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.697889 6304 tablet_service.cc:1467] Processing CreateTablet for tablet fb2fec4b506c4846935f68b456479e1d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 14:04:56.675997 6512 tablet_bootstrap.cc:654] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:56.707388 6513 tablet_bootstrap.cc:654] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:56.710489 6383 tablet_service.cc:1467] Processing CreateTablet for tablet c4ee8e66a5b74a4394a4e339a0af957d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240503 14:04:56.711908 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ee8e66a5b74a4394a4e339a0af957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.700139 6460 tablet_service.cc:1467] Processing CreateTablet for tablet fb2fec4b506c4846935f68b456479e1d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 14:04:56.713615 6386 tablet_service.cc:1467] Processing CreateTablet for tablet a4080dad66944ea291f6d69736cd2c23 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240503 14:04:56.718382 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4080dad66944ea291f6d69736cd2c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.715801 6388 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0fcc21be834ead9ab016a33b25928c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240503 14:04:56.721285 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0fcc21be834ead9ab016a33b25928c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.723287 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6985cc0483c34fcd95b3e56b8b59bfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.716917 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 0e49c03a6cf8400b98ce8d56e55089e1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240503 14:04:56.731235 6379 tablet_service.cc:1467] Processing CreateTablet for tablet 66d5e8c5c12a43479ecc3382c9576c60 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240503 14:04:56.736493 6306 tablet_service.cc:1467] Processing CreateTablet for tablet 0afc2b5c1e62414587bea6a93f52c0c9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 14:04:56.737893 6305 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2f4c7143ea4d49b2be0871738a9a50 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 14:04:56.744333 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 0afc2b5c1e62414587bea6a93f52c0c9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 14:04:56.758663 6307 tablet_service.cc:1467] Processing CreateTablet for tablet e17974ef0de7408c9a2fcebd1f42e07b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 14:04:56.749001 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d5e8c5c12a43479ecc3382c9576c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.763108 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2fec4b506c4846935f68b456479e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.787509 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e49c03a6cf8400b98ce8d56e55089e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.790522 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2f4c7143ea4d49b2be0871738a9a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.791241 6383 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2f4c7143ea4d49b2be0871738a9a50 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 14:04:56.796418 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afc2b5c1e62414587bea6a93f52c0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.797870 6513 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:56.774621 6514 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:56.800426 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4080dad66944ea291f6d69736cd2c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.800808 6514 ts_tablet_manager.cc:1367] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.150s user 0.010s sys 0.003s
I20240503 14:04:56.803787 6514 raft_consensus.cc:348] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.804721 6514 raft_consensus.cc:374] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:56.805085 6514 raft_consensus.cc:729] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:56.806015 6514 consensus_queue.cc:260] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.806864 6308 tablet_service.cc:1467] Processing CreateTablet for tablet a315c69a72a240f0a5be1b1b8f3625e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 14:04:56.807862 6513 ts_tablet_manager.cc:1367] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.011s
I20240503 14:04:56.776786 6386 tablet_service.cc:1467] Processing CreateTablet for tablet 0e49c03a6cf8400b98ce8d56e55089e1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240503 14:04:56.808590 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2fec4b506c4846935f68b456479e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.776932 6456 tablet_service.cc:1467] Processing CreateTablet for tablet 0e49c03a6cf8400b98ce8d56e55089e1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
W20240503 14:04:56.779868 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:56.779642 6382 tablet_service.cc:1467] Processing CreateTablet for tablet 0afc2b5c1e62414587bea6a93f52c0c9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240503 14:04:56.798763 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d5e8c5c12a43479ecc3382c9576c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.810707 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e49c03a6cf8400b98ce8d56e55089e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.811300 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afc2b5c1e62414587bea6a93f52c0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.770864 6387 tablet_service.cc:1467] Processing CreateTablet for tablet fb2fec4b506c4846935f68b456479e1d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240503 14:04:56.812826 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2fec4b506c4846935f68b456479e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.774730 6309 tablet_service.cc:1467] Processing CreateTablet for tablet 157836a198e44425a9f455d4c07fa5c4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 14:04:56.816402 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2f4c7143ea4d49b2be0871738a9a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.816764 6512 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:56.817263 6512 ts_tablet_manager.cc:1367] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.154s user 0.013s sys 0.009s
I20240503 14:04:56.819969 6512 raft_consensus.cc:348] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.820923 6512 raft_consensus.cc:374] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:56.821267 6512 raft_consensus.cc:729] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:56.822127 6512 consensus_queue.cc:260] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.825202 6313 tablet_service.cc:1467] Processing CreateTablet for tablet df77168e3be74202b705c73539c514d0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 14:04:56.825696 6457 tablet_service.cc:1467] Processing CreateTablet for tablet 4a2f4c7143ea4d49b2be0871738a9a50 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240503 14:04:56.829664 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e49c03a6cf8400b98ce8d56e55089e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.832429 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0afc2b5c1e62414587bea6a93f52c0c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.835688 6380 tablet_service.cc:1467] Processing CreateTablet for tablet e17974ef0de7408c9a2fcebd1f42e07b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 14:04:56.837153 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17974ef0de7408c9a2fcebd1f42e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.840351 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a315c69a72a240f0a5be1b1b8f3625e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.842573 6513 raft_consensus.cc:348] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.846765 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17974ef0de7408c9a2fcebd1f42e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.848630 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a2f4c7143ea4d49b2be0871738a9a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.852715 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df77168e3be74202b705c73539c514d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.856431 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157836a198e44425a9f455d4c07fa5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.861162 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 157836a198e44425a9f455d4c07fa5c4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 14:04:56.862524 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157836a198e44425a9f455d4c07fa5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.849268 6513 raft_consensus.cc:374] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:56.872182 6513 raft_consensus.cc:729] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:56.873737 6348 heartbeater.cc:503] Master 127.4.250.254:36245 was elected leader, sending a full tablet report...
I20240503 14:04:56.846801 6310 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d57a792584b2ca83a4a8747a1f1c5 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 14:04:56.878571 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d57a792584b2ca83a4a8747a1f1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.878168 6513 consensus_queue.cc:260] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.881659 6459 tablet_service.cc:1467] Processing CreateTablet for tablet a315c69a72a240f0a5be1b1b8f3625e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 14:04:56.884202 6459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a315c69a72a240f0a5be1b1b8f3625e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.889223 6384 tablet_service.cc:1467] Processing CreateTablet for tablet df77168e3be74202b705c73539c514d0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 14:04:56.889966 6512 ts_tablet_manager.cc:1398] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.072s user 0.007s sys 0.004s
I20240503 14:04:56.890630 6384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df77168e3be74202b705c73539c514d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.891095 6512 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:56.889225 6386 tablet_service.cc:1467] Processing CreateTablet for tablet a315c69a72a240f0a5be1b1b8f3625e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240503 14:04:56.897085 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a315c69a72a240f0a5be1b1b8f3625e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.902387 6382 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d57a792584b2ca83a4a8747a1f1c5 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 14:04:56.903887 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d57a792584b2ca83a4a8747a1f1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.924968 6381 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0562bcbd0a407a9c73885ac89858f9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 14:04:56.926379 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0562bcbd0a407a9c73885ac89858f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.905206 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0562bcbd0a407a9c73885ac89858f9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 14:04:56.906877 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 4148780059584da6a02aaedbf2dd9cf0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 14:04:56.913324 6306 tablet_service.cc:1467] Processing CreateTablet for tablet 2b30b12892024636b275c9070901327b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 14:04:56.917460 6456 tablet_service.cc:1467] Processing CreateTablet for tablet fe2d57a792584b2ca83a4a8747a1f1c5 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240503 14:04:56.929009 6388 tablet_service.cc:1467] Processing CreateTablet for tablet 157836a198e44425a9f455d4c07fa5c4 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240503 14:04:56.930383 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2d57a792584b2ca83a4a8747a1f1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.934494 6512 tablet_bootstrap.cc:654] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:56.919536 6387 tablet_service.cc:1467] Processing CreateTablet for tablet 4148780059584da6a02aaedbf2dd9cf0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 14:04:56.920704 6453 tablet_service.cc:1467] Processing CreateTablet for tablet e17974ef0de7408c9a2fcebd1f42e07b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240503 14:04:56.936987 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17974ef0de7408c9a2fcebd1f42e07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.942749 6455 tablet_service.cc:1467] Processing CreateTablet for tablet 4148780059584da6a02aaedbf2dd9cf0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240503 14:04:56.930485 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157836a198e44425a9f455d4c07fa5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.956303 6497 heartbeater.cc:503] Master 127.4.250.254:36245 was elected leader, sending a full tablet report...
I20240503 14:04:56.909817 6462 tablet_service.cc:1467] Processing CreateTablet for tablet df77168e3be74202b705c73539c514d0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240503 14:04:56.962378 6520 raft_consensus.cc:482] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:56.962507 6423 heartbeater.cc:503] Master 127.4.250.254:36245 was elected leader, sending a full tablet report...
I20240503 14:04:56.963625 6513 ts_tablet_manager.cc:1398] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.124s user 0.002s sys 0.011s
I20240503 14:04:56.943113 6386 tablet_service.cc:1467] Processing CreateTablet for tablet 2b30b12892024636b275c9070901327b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 14:04:56.962050 6514 ts_tablet_manager.cc:1398] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.161s user 0.016s sys 0.017s
I20240503 14:04:56.973135 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4148780059584da6a02aaedbf2dd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:56.974921 6520 raft_consensus.cc:504] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:56.975982 6461 tablet_service.cc:1467] Processing CreateTablet for tablet 2b30b12892024636b275c9070901327b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240503 14:04:56.929041 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0562bcbd0a407a9c73885ac89858f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.002506 6513 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:56.954103 6305 tablet_service.cc:1467] Processing CreateTablet for tablet b0285fa210834b8c9b116d0cf7bb2430 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240503 14:04:57.008910 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df77168e3be74202b705c73539c514d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.009152 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4148780059584da6a02aaedbf2dd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.012296 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b30b12892024636b275c9070901327b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.030197 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4148780059584da6a02aaedbf2dd9cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.012538 6514 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:57.033910 6513 tablet_bootstrap.cc:654] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.017424 6460 tablet_service.cc:1467] Processing CreateTablet for tablet 5f401553cb154845b5d0a890810a9837 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 14:04:57.036478 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0285fa210834b8c9b116d0cf7bb2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.037232 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f401553cb154845b5d0a890810a9837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.039037 6520 leader_election.cc:288] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.040023 6385 tablet_service.cc:1467] Processing CreateTablet for tablet b0285fa210834b8c9b116d0cf7bb2430 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240503 14:04:57.041363 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0285fa210834b8c9b116d0cf7bb2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.040045 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b30b12892024636b275c9070901327b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.046512 6514 tablet_bootstrap.cc:654] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:56.990664 6456 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0562bcbd0a407a9c73885ac89858f9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240503 14:04:57.022348 6304 tablet_service.cc:1467] Processing CreateTablet for tablet 2e09030e41364fbaadad3e4ea90d6166 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240503 14:04:57.053668 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e09030e41364fbaadad3e4ea90d6166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.056023 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0562bcbd0a407a9c73885ac89858f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.048934 6457 tablet_service.cc:1467] Processing CreateTablet for tablet a707858e929f40f89e2be68639a03040 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 14:04:57.019318 6308 tablet_service.cc:1467] Processing CreateTablet for tablet a707858e929f40f89e2be68639a03040 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 14:04:57.060075 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a707858e929f40f89e2be68639a03040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.011581 6458 tablet_service.cc:1467] Processing CreateTablet for tablet b0285fa210834b8c9b116d0cf7bb2430 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240503 14:04:57.092003 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a707858e929f40f89e2be68639a03040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.092816 6383 tablet_service.cc:1467] Processing CreateTablet for tablet 5f401553cb154845b5d0a890810a9837 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 14:04:57.094157 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f401553cb154845b5d0a890810a9837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.094308 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd4c415b34640bd82515afe5d3a7632 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 14:04:57.095600 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd4c415b34640bd82515afe5d3a7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.106112 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 2e09030e41364fbaadad3e4ea90d6166 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240503 14:04:57.106561 6379 tablet_service.cc:1467] Processing CreateTablet for tablet a707858e929f40f89e2be68639a03040 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240503 14:04:57.107805 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a707858e929f40f89e2be68639a03040. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.109926 6313 tablet_service.cc:1467] Processing CreateTablet for tablet e09711da83964cb1b409403e16852e17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240503 14:04:57.111193 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09711da83964cb1b409403e16852e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.120410 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b30b12892024636b275c9070901327b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.121330 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e09030e41364fbaadad3e4ea90d6166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.129420 6518 raft_consensus.cc:482] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.129976 6518 raft_consensus.cc:504] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.136324 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0285fa210834b8c9b116d0cf7bb2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.146490 6519 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.147017 6519 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.149525 6519 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.134228 6518 leader_election.cc:288] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:57.139951 6310 tablet_service.cc:1467] Processing CreateTablet for tablet 5f401553cb154845b5d0a890810a9837 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240503 14:04:57.169590 6309 tablet_service.cc:1467] Processing CreateTablet for tablet 6741d371ee4e47b39f1c444c9fb8fe8a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 14:04:57.170997 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6741d371ee4e47b39f1c444c9fb8fe8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.185206 6513 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:57.185791 6513 ts_tablet_manager.cc:1367] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.187s user 0.010s sys 0.008s
I20240503 14:04:57.230304 6455 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd4c415b34640bd82515afe5d3a7632 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 14:04:57.231284 6379 tablet_service.cc:1467] Processing CreateTablet for tablet 2e09030e41364fbaadad3e4ea90d6166 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240503 14:04:57.232031 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f401553cb154845b5d0a890810a9837. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.232585 6379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e09030e41364fbaadad3e4ea90d6166. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.238569 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:57.238399 6513 raft_consensus.cc:348] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.239526 6513 raft_consensus.cc:374] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:04:57.240463 6364 leader_election.cc:341] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.243063 6384 tablet_service.cc:1467] Processing CreateTablet for tablet 1fd4c415b34640bd82515afe5d3a7632 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240503 14:04:57.245352 6382 tablet_service.cc:1467] Processing CreateTablet for tablet e09711da83964cb1b409403e16852e17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240503 14:04:57.252472 6512 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:57.254024 6512 ts_tablet_manager.cc:1367] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.363s user 0.015s sys 0.019s
I20240503 14:04:57.256731 6386 tablet_service.cc:1467] Processing CreateTablet for tablet 6741d371ee4e47b39f1c444c9fb8fe8a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 14:04:57.239894 6513 raft_consensus.cc:729] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.257524 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:57.231665 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd4c415b34640bd82515afe5d3a7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.255578 6381 tablet_service.cc:1467] Processing CreateTablet for tablet 1658a580bb164365a9df9fed7a641de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
W20240503 14:04:57.268780 6362 leader_election.cc:341] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.270417 6362 leader_election.cc:302] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:57.271834 6521 raft_consensus.cc:2726] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.282088 6380 tablet_service.cc:1467] Processing CreateTablet for tablet c9b840e10839487fac480c2bbf79f329 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 14:04:57.283097 6383 tablet_service.cc:1467] Processing CreateTablet for tablet f627e726d194488cb11419bbff15aa2d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 14:04:57.275601 6513 consensus_queue.cc:260] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.285583 6513 ts_tablet_manager.cc:1398] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
I20240503 14:04:57.285696 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:04:57.287530 6435 leader_election.cc:341] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.278033 6384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fd4c415b34640bd82515afe5d3a7632. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.288959 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f627e726d194488cb11419bbff15aa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.294005 6512 raft_consensus.cc:348] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.294997 6512 raft_consensus.cc:374] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.295337 6512 raft_consensus.cc:729] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.294126 6387 tablet_service.cc:1467] Processing CreateTablet for tablet 8587555ef5ac4e31b7ab986f265fd3d7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240503 14:04:57.304309 6514 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:57.304946 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6741d371ee4e47b39f1c444c9fb8fe8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.305006 6514 ts_tablet_manager.cc:1367] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.293s user 0.009s sys 0.010s
I20240503 14:04:57.309563 6514 raft_consensus.cc:348] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.314924 6514 raft_consensus.cc:374] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.315383 6514 raft_consensus.cc:729] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.312448 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587555ef5ac4e31b7ab986f265fd3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.312868 6512 consensus_queue.cc:260] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.321179 6519 raft_consensus.cc:482] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.321746 6519 raft_consensus.cc:504] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.321682 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658a580bb164365a9df9fed7a641de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.333467 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09711da83964cb1b409403e16852e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.338557 6519 leader_election.cc:288] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.343922 6312 tablet_service.cc:1467] Processing CreateTablet for tablet c9b840e10839487fac480c2bbf79f329 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 14:04:57.345335 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b840e10839487fac480c2bbf79f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.347175 6383 tablet_service.cc:1467] Processing CreateTablet for tablet a594e34044584aa09a50d15954cbb81b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 14:04:57.350960 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 8587555ef5ac4e31b7ab986f265fd3d7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240503 14:04:57.355343 6308 tablet_service.cc:1467] Processing CreateTablet for tablet a2af980c77644cb8af111aa4f519f173 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 14:04:57.356315 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587555ef5ac4e31b7ab986f265fd3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.356997 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2af980c77644cb8af111aa4f519f173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.352846 6306 tablet_service.cc:1467] Processing CreateTablet for tablet 41d1828e46eb4868900cf3ec7e69b5b0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240503 14:04:57.361294 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d1828e46eb4868900cf3ec7e69b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.353137 6311 tablet_service.cc:1467] Processing CreateTablet for tablet a594e34044584aa09a50d15954cbb81b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 14:04:57.365898 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a594e34044584aa09a50d15954cbb81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.354647 6388 tablet_service.cc:1467] Processing CreateTablet for tablet a2af980c77644cb8af111aa4f519f173 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 14:04:57.369288 6305 tablet_service.cc:1467] Processing CreateTablet for tablet 1658a580bb164365a9df9fed7a641de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240503 14:04:57.371376 6513 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:57.384519 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b840e10839487fac480c2bbf79f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.386410 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:57.377116 6461 tablet_service.cc:1467] Processing CreateTablet for tablet e09711da83964cb1b409403e16852e17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240503 14:04:57.397516 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09711da83964cb1b409403e16852e17. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:04:57.398231 6286 leader_election.cc:341] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.400399 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:57.400990 6385 tablet_service.cc:1467] Processing CreateTablet for tablet 41d1828e46eb4868900cf3ec7e69b5b0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
W20240503 14:04:57.402493 6286 leader_election.cc:341] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.400732 6305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658a580bb164365a9df9fed7a641de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.418424 6383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a594e34044584aa09a50d15954cbb81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.427165 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d1828e46eb4868900cf3ec7e69b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.522568 6462 tablet_service.cc:1467] Processing CreateTablet for tablet c9b840e10839487fac480c2bbf79f329 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240503 14:04:57.524022 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b840e10839487fac480c2bbf79f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.525852 6456 tablet_service.cc:1467] Processing CreateTablet for tablet f627e726d194488cb11419bbff15aa2d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 14:04:57.529721 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f627e726d194488cb11419bbff15aa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.411334 6514 consensus_queue.cc:260] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.536504 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 8587555ef5ac4e31b7ab986f265fd3d7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240503 14:04:57.537966 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8587555ef5ac4e31b7ab986f265fd3d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.539769 6514 ts_tablet_manager.cc:1398] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.234s user 0.007s sys 0.001s
I20240503 14:04:57.540916 6514 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:57.552558 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 1658a580bb164365a9df9fed7a641de0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240503 14:04:57.552875 6455 tablet_service.cc:1467] Processing CreateTablet for tablet 41d1828e46eb4868900cf3ec7e69b5b0 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240503 14:04:57.553225 6460 tablet_service.cc:1467] Processing CreateTablet for tablet a594e34044584aa09a50d15954cbb81b (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240503 14:04:57.553956 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1658a580bb164365a9df9fed7a641de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.554639 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a594e34044584aa09a50d15954cbb81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.555613 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2af980c77644cb8af111aa4f519f173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.557389 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:57.557415 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d1828e46eb4868900cf3ec7e69b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.561122 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:57.565917 6289 leader_election.cc:341] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.566442 6289 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
W20240503 14:04:57.573009 6289 leader_election.cc:341] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.573552 6289 leader_election.cc:302] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:57.572991 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:57.576165 6519 raft_consensus.cc:2726] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.576840 6519 raft_consensus.cc:2726] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.591643 6512 ts_tablet_manager.cc:1398] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.337s user 0.038s sys 0.028s
I20240503 14:04:57.593147 6512 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:57.600697 6457 tablet_service.cc:1467] Processing CreateTablet for tablet 6741d371ee4e47b39f1c444c9fb8fe8a (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240503 14:04:57.602178 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6741d371ee4e47b39f1c444c9fb8fe8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.605646 6512 tablet_bootstrap.cc:654] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.610668 6514 tablet_bootstrap.cc:654] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.625795 6513 tablet_bootstrap.cc:654] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.818115 6520 raft_consensus.cc:482] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.818710 6520 raft_consensus.cc:504] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.828467 6520 leader_election.cc:288] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.830304 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:57.831652 6362 leader_election.cc:341] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.833526 6512 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:57.834122 6513 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:57.834201 6512 ts_tablet_manager.cc:1367] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.241s user 0.078s sys 0.084s
I20240503 14:04:57.834841 6513 ts_tablet_manager.cc:1367] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.464s user 0.011s sys 0.012s
I20240503 14:04:57.835826 6514 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:57.836938 6514 ts_tablet_manager.cc:1367] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.296s user 0.016s sys 0.006s
I20240503 14:04:57.837330 6512 raft_consensus.cc:348] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.838160 6512 raft_consensus.cc:374] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.837863 6513 raft_consensus.cc:348] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.838492 6512 raft_consensus.cc:729] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.838908 6513 raft_consensus.cc:374] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.839275 6513 raft_consensus.cc:729] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.839437 6512 consensus_queue.cc:260] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.839737 6514 raft_consensus.cc:348] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.840641 6514 raft_consensus.cc:374] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.840987 6514 raft_consensus.cc:729] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.840070 6513 consensus_queue.cc:260] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.841637 6512 ts_tablet_manager.cc:1398] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:04:57.841663 6514 consensus_queue.cc:260] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.843017 6512 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:57.843236 6513 ts_tablet_manager.cc:1398] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:04:57.843631 6520 raft_consensus.cc:482] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.844288 6513 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:57.844143 6520 raft_consensus.cc:504] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.844358 6514 ts_tablet_manager.cc:1398] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20240503 14:04:57.845638 6514 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:57.846323 6520 leader_election.cc:288] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.848449 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:57.849187 6323 raft_consensus.cc:2454] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:04:57.850723 6513 tablet_bootstrap.cc:654] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.850678 6362 leader_election.cc:302] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:57.851675 6514 tablet_bootstrap.cc:654] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.851675 6520 raft_consensus.cc:2781] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:57.852267 6520 raft_consensus.cc:482] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:57.853996 6520 raft_consensus.cc:3037] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:04:57.862342 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:57.874620 6512 tablet_bootstrap.cc:654] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.879169 6520 raft_consensus.cc:504] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.881435 6520 leader_election.cc:288] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.883145 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:57.883961 6323 raft_consensus.cc:3037] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:57.892743 6323 raft_consensus.cc:2454] T 1fd965edad81421d8b03f93895f80ebe P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:04:57.893494 6514 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:57.894061 6514 ts_tablet_manager.cc:1367] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.049s user 0.025s sys 0.019s
I20240503 14:04:57.894549 6513 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:57.894301 6362 leader_election.cc:302] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:57.895063 6513 ts_tablet_manager.cc:1367] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.015s
I20240503 14:04:57.895427 6520 raft_consensus.cc:2781] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:57.371806 6313 tablet_service.cc:1467] Processing CreateTablet for tablet f627e726d194488cb11419bbff15aa2d (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240503 14:04:57.897306 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f627e726d194488cb11419bbff15aa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.896983 6514 raft_consensus.cc:348] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.897790 6514 raft_consensus.cc:374] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.898243 6514 raft_consensus.cc:729] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.897768 6513 raft_consensus.cc:348] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.898747 6513 raft_consensus.cc:374] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.899057 6513 raft_consensus.cc:729] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.899225 6514 consensus_queue.cc:260] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.899940 6513 consensus_queue.cc:260] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.902359 6537 raft_consensus.cc:482] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.902530 6514 ts_tablet_manager.cc:1398] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240503 14:04:57.902868 6537 raft_consensus.cc:504] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.903744 6514 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:57.905267 6537 leader_election.cc:288] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:57.907141 6381 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6c0c9a9554a998f3efbacf240ca17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 14:04:57.908972 6512 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:57.909510 6512 ts_tablet_manager.cc:1367] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.000s
I20240503 14:04:57.910369 6514 tablet_bootstrap.cc:654] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:57.910589 6436 leader_election.cc:341] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.911064 6436 leader_election.cc:302] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:57.912513 6512 raft_consensus.cc:348] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.913394 6512 raft_consensus.cc:374] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:57.913731 6512 raft_consensus.cc:729] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:57.914872 6512 consensus_queue.cc:260] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.916350 6513 ts_tablet_manager.cc:1398] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.021s user 0.011s sys 0.001s
I20240503 14:04:57.917399 6512 ts_tablet_manager.cc:1398] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240503 14:04:57.917505 6513 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:57.916275 6518 raft_consensus.cc:2726] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.909138 6380 tablet_service.cc:1467] Processing CreateTablet for tablet 3586823a1bd546f08b8007d4b55ef43e (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20240503 14:04:57.921977 6306 consensus_meta.cc:287] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent flushing consensus metadata: real 0.532s user 0.009s sys 0.004s
I20240503 14:04:57.919525 6520 raft_consensus.cc:686] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:04:57.924188 6461 tablet_service.cc:1467] Processing CreateTablet for tablet a2af980c77644cb8af111aa4f519f173 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240503 14:04:57.920392 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6c0c9a9554a998f3efbacf240ca17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.925546 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3586823a1bd546f08b8007d4b55ef43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.925737 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2af980c77644cb8af111aa4f519f173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.927235 6520 consensus_queue.cc:237] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.928690 6454 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6c0c9a9554a998f3efbacf240ca17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 14:04:57.930009 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6c0c9a9554a998f3efbacf240ca17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:57.938079 6512 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:57.944010 6513 tablet_bootstrap.cc:654] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.947917 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d070812c38ea4aeda6ce94fac9ed7de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:57.948789 6472 raft_consensus.cc:2454] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:57.951978 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:57.953580 6512 tablet_bootstrap.cc:654] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:57.960352 6289 leader_election.cc:302] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:57.961318 6537 raft_consensus.cc:2781] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:57.961776 6537 raft_consensus.cc:482] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:57.962136 6537 raft_consensus.cc:3037] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:57.964536 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:57.964728 6545 raft_consensus.cc:482] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.965459 6545 raft_consensus.cc:504] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.967675 6545 leader_election.cc:288] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:04:57.961453 6364 leader_election.cc:341] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.969661 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:57.970568 6364 leader_election.cc:341] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.971015 6364 leader_election.cc:302] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
W20240503 14:04:57.971521 6362 leader_election.cc:341] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.972332 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:57.976382 6518 raft_consensus.cc:482] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:57.976886 6518 raft_consensus.cc:504] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:57.978868 6518 leader_election.cc:288] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
W20240503 14:04:57.992748 6364 leader_election.cc:341] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:57.992936 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:57.993376 6364 leader_election.cc:302] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:57.994374 6544 raft_consensus.cc:2726] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.994992 6544 raft_consensus.cc:2726] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:57.996927 6462 tablet_service.cc:1467] Processing CreateTablet for tablet 3586823a1bd546f08b8007d4b55ef43e (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240503 14:04:57.998435 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3586823a1bd546f08b8007d4b55ef43e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:04:58.007573 6436 leader_election.cc:341] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.014925 6385 tablet_service.cc:1467] Processing CreateTablet for tablet 88ac2eddffd74f50bc6b87f3a7c0827c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 14:04:58.018025 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ac2eddffd74f50bc6b87f3a7c0827c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.018370 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d070812c38ea4aeda6ce94fac9ed7de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:04:58.025945 6286 leader_election.cc:341] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.027773 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 5da6c0c9a9554a998f3efbacf240ca17 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240503 14:04:58.029162 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5da6c0c9a9554a998f3efbacf240ca17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.033197 6310 tablet_service.cc:1467] Processing CreateTablet for tablet 3586823a1bd546f08b8007d4b55ef43e (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240503 14:04:58.034613 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3586823a1bd546f08b8007d4b55ef43e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:04:58.034966 6309 consensus_meta.cc:287] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent flushing consensus metadata: real 0.733s user 0.007s sys 0.003s
I20240503 14:04:58.033341 6313 tablet_service.cc:1467] Processing CreateTablet for tablet 88ac2eddffd74f50bc6b87f3a7c0827c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 14:04:58.045178 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ac2eddffd74f50bc6b87f3a7c0827c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.046882 6458 tablet_service.cc:1467] Processing CreateTablet for tablet 88ac2eddffd74f50bc6b87f3a7c0827c (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240503 14:04:58.048377 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ac2eddffd74f50bc6b87f3a7c0827c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.048951 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:58.050792 6364 leader_election.cc:341] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.062176 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:04:58.081488 6498 tablet.cc:2369] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:04:58.088882 6435 leader_election.cc:341] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.089388 6435 leader_election.cc:302] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.092126 6518 raft_consensus.cc:2726] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:04:58.092542 6387 consensus_meta.cc:287] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent flushing consensus metadata: real 0.642s user 0.014s sys 0.002s
I20240503 14:04:58.094180 6518 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.094642 6518 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:04:58.096477 6386 consensus_meta.cc:287] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: Time spent flushing consensus metadata: real 0.724s user 0.010s sys 0.000s
I20240503 14:04:58.096635 6518 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.098373 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.099053 6323 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:58.100320 6436 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:58.109690 6518 raft_consensus.cc:2781] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.110097 6518 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.110430 6518 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.111550 6513 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.112033 6513 ts_tablet_manager.cc:1367] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.195s user 0.014s sys 0.011s
I20240503 14:04:58.114784 6513 raft_consensus.cc:348] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.115600 6513 raft_consensus.cc:374] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.114730 6537 raft_consensus.cc:504] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.116775 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:04:58.118124 6435 leader_election.cc:341] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.118783 6512 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:58.119248 6512 ts_tablet_manager.cc:1367] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.181s user 0.034s sys 0.049s
I20240503 14:04:58.121147 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d070812c38ea4aeda6ce94fac9ed7de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:04:58.124336 6286 leader_election.cc:341] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.125090 6537 leader_election.cc:288] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.121971 6518 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.128484 6518 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.128785 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:58.127393 6454 tablet_service.cc:1467] Processing CreateTablet for tablet c7b3d6ea1b264b0e8781218ca4ee5155 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 14:04:58.129139 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:58.130194 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b3d6ea1b264b0e8781218ca4ee5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.129359 6323 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:04:58.130592 6435 leader_election.cc:341] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.133347 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 9477869b1b604feb8f56531957fb13e6 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 14:04:58.133589 6309 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 14:04:58.134660 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9477869b1b604feb8f56531957fb13e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.130327 6385 tablet_service.cc:1467] Processing CreateTablet for tablet 9477869b1b604feb8f56531957fb13e6 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 14:04:58.136413 6385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9477869b1b604feb8f56531957fb13e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.123176 6514 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.123090 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d070812c38ea4aeda6ce94fac9ed7de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:58.140525 6471 raft_consensus.cc:3037] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.140916 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.115898 6513 raft_consensus.cc:729] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.144685 6513 consensus_queue.cc:260] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.146000 6457 tablet_service.cc:1467] Processing CreateTablet for tablet 9477869b1b604feb8f56531957fb13e6 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240503 14:04:58.146905 6513 ts_tablet_manager.cc:1398] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20240503 14:04:58.147326 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9477869b1b604feb8f56531957fb13e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.147897 6513 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.150362 6514 ts_tablet_manager.cc:1367] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.247s user 0.021s sys 0.012s
I20240503 14:04:58.130331 6386 tablet_service.cc:1467] Processing CreateTablet for tablet c7b3d6ea1b264b0e8781218ca4ee5155 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 14:04:58.153393 6513 tablet_bootstrap.cc:654] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.155907 6380 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 14:04:58.153326 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b3d6ea1b264b0e8781218ca4ee5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.157878 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.160796 6323 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:58.153829 6471 raft_consensus.cc:2454] T d070812c38ea4aeda6ce94fac9ed7de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:04:58.162088 6436 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.162412 6514 raft_consensus.cc:348] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.163199 6514 raft_consensus.cc:374] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.163487 6514 raft_consensus.cc:729] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.168666 6518 raft_consensus.cc:2781] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.169693 6289 leader_election.cc:302] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.170537 6537 raft_consensus.cc:2781] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.170441 6514 consensus_queue.cc:260] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.173264 6313 tablet_service.cc:1467] Processing CreateTablet for tablet c7b3d6ea1b264b0e8781218ca4ee5155 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240503 14:04:58.174820 6313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7b3d6ea1b264b0e8781218ca4ee5155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.178980 6458 tablet_service.cc:1467] Processing CreateTablet for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240503 14:04:58.186374 6307 tablet_service.cc:1467] Processing CreateTablet for tablet a19868aca07442c4bc78952934029bb9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 14:04:58.186903 6387 tablet_service.cc:1467] Processing CreateTablet for tablet a19868aca07442c4bc78952934029bb9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 14:04:58.188047 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19868aca07442c4bc78952934029bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.187687 6454 tablet_service.cc:1467] Processing CreateTablet for tablet a19868aca07442c4bc78952934029bb9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240503 14:04:58.188378 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19868aca07442c4bc78952934029bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.189406 6310 tablet_service.cc:1467] Processing CreateTablet for tablet 1407f54e751147b3b05f2b4083dbc041 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 14:04:58.192142 6310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1407f54e751147b3b05f2b4083dbc041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.192293 6461 tablet_service.cc:1467] Processing CreateTablet for tablet 1407f54e751147b3b05f2b4083dbc041 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 14:04:58.193840 6388 tablet_service.cc:1467] Processing CreateTablet for tablet 1407f54e751147b3b05f2b4083dbc041 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240503 14:04:58.195928 6388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1407f54e751147b3b05f2b4083dbc041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.197701 6220 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40820 (request call id 2) took 1173 ms. Trace:
I20240503 14:04:58.198113 6220 rpcz_store.cc:276] 0503 14:04:57.023851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:04:57.030713 (+ 6862us) service_pool.cc:225] Handling call
0503 14:04:58.197626 (+1166913us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":156499,"reported_tablets":24}
I20240503 14:04:58.121904 6512 raft_consensus.cc:348] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.199504 6512 raft_consensus.cc:374] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.199815 6512 raft_consensus.cc:729] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.200668 6512 consensus_queue.cc:260] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.194234 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.208115 6454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19868aca07442c4bc78952934029bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.208169 6219 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:40828 (request call id 2) took 1061 ms. Trace:
I20240503 14:04:58.208640 6219 rpcz_store.cc:276] 0503 14:04:57.146449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:04:57.146581 (+ 132us) service_pool.cc:225] Handling call
0503 14:04:58.207955 (+1061374us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":603615,"reported_tablets":22}
I20240503 14:04:58.211159 6537 raft_consensus.cc:686] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:04:58.211906 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1407f54e751147b3b05f2b4083dbc041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.221863 6537 consensus_queue.cc:237] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.214291 6312 tablet_service.cc:1467] Processing CreateTablet for tablet bf848b599cbc45ecb8772ecfb50276e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 14:04:58.224324 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf848b599cbc45ecb8772ecfb50276e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.216562 6518 raft_consensus.cc:686] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:58.219559 6380 tablet_service.cc:1467] Processing CreateTablet for tablet bf848b599cbc45ecb8772ecfb50276e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 14:04:58.233081 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf848b599cbc45ecb8772ecfb50276e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.211058 6221 catalog_manager.cc:5484] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:58.220505 6514 ts_tablet_manager.cc:1398] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.070s user 0.016s sys 0.008s
I20240503 14:04:58.234845 6514 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.232796 6518 consensus_queue.cc:237] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.241003 6552 raft_consensus.cc:482] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.241518 6306 tablet_service.cc:1467] Processing CreateTablet for tablet f5ce484e160849488b2637359d07ba88 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 14:04:58.242631 6456 tablet_service.cc:1467] Processing CreateTablet for tablet bf848b599cbc45ecb8772ecfb50276e7 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240503 14:04:58.243225 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ce484e160849488b2637359d07ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.243866 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf848b599cbc45ecb8772ecfb50276e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.241513 6552 raft_consensus.cc:504] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.261962 6387 tablet_service.cc:1467] Processing CreateTablet for tablet f5ce484e160849488b2637359d07ba88 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 14:04:58.263331 6462 tablet_service.cc:1467] Processing CreateTablet for tablet f5ce484e160849488b2637359d07ba88 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240503 14:04:58.269538 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.270489 6457 tablet_service.cc:1467] Processing CreateTablet for tablet 665b49a99fb74a1ab8ad655e76ca0e34 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20240503 14:04:58.272014 6286 leader_election.cc:341] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.272043 6387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ce484e160849488b2637359d07ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.272059 6381 tablet_service.cc:1467] Processing CreateTablet for tablet 665b49a99fb74a1ab8ad655e76ca0e34 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240503 14:04:58.273798 6381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b49a99fb74a1ab8ad655e76ca0e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.273499 6308 tablet_service.cc:1467] Processing CreateTablet for tablet 665b49a99fb74a1ab8ad655e76ca0e34 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240503 14:04:58.269908 6552 leader_election.cc:288] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.275563 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:58.278671 6289 leader_election.cc:341] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.279145 6289 leader_election.cc:302] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.280558 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b49a99fb74a1ab8ad655e76ca0e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.282188 6308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b49a99fb74a1ab8ad655e76ca0e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.285576 6462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ce484e160849488b2637359d07ba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.279336 6513 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.280647 6552 raft_consensus.cc:2726] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:58.295006 6514 tablet_bootstrap.cc:654] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.301450 6460 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 14:04:58.302063 6512 ts_tablet_manager.cc:1398] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.183s user 0.009s sys 0.001s
I20240503 14:04:58.301873 6382 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 14:04:58.303028 6460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.303177 6512 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:58.303028 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.304015 6307 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240503 14:04:58.305598 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc21dfebd854cb3b4f5ce3fe803b0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.292007 6513 ts_tablet_manager.cc:1367] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.144s user 0.013s sys 0.007s
I20240503 14:04:58.298811 6455 tablet_service.cc:1467] Processing CreateTablet for tablet e75042082bf74978bd28f824f5bee4e9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 14:04:58.308956 6455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75042082bf74978bd28f824f5bee4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.332029 6513 raft_consensus.cc:348] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.333058 6513 raft_consensus.cc:374] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.333586 6513 raft_consensus.cc:729] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.318733 6219 catalog_manager.cc:5484] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:58.336795 6513 consensus_queue.cc:260] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.341146 6513 ts_tablet_manager.cc:1398] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.024s user 0.003s sys 0.006s
I20240503 14:04:58.342468 6513 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.345058 6386 tablet_service.cc:1467] Processing CreateTablet for tablet e75042082bf74978bd28f824f5bee4e9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 14:04:58.346836 6386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75042082bf74978bd28f824f5bee4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.352895 6513 tablet_bootstrap.cc:654] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.357041 6309 tablet_service.cc:1467] Processing CreateTablet for tablet e75042082bf74978bd28f824f5bee4e9 (DEFAULT_TABLE table=test-workload [id=5962ac28e0864fefbed28c4368b99536]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240503 14:04:58.366727 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75042082bf74978bd28f824f5bee4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:04:58.362700 6512 tablet_bootstrap.cc:654] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.381413 6520 consensus_queue.cc:1035] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:58.398682 6513 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.402207 6513 ts_tablet_manager.cc:1367] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.011s
I20240503 14:04:58.407279 6513 raft_consensus.cc:348] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.414228 6513 raft_consensus.cc:374] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.423460 6513 raft_consensus.cc:729] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.429416 6513 consensus_queue.cc:260] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.432394 6513 ts_tablet_manager.cc:1398] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20240503 14:04:58.440379 6513 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.441835 6514 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.442473 6514 ts_tablet_manager.cc:1367] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.208s user 0.018s sys 0.001s
I20240503 14:04:58.448669 6544 raft_consensus.cc:482] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.449353 6544 raft_consensus.cc:504] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.447301 6514 raft_consensus.cc:348] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.453022 6514 raft_consensus.cc:374] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.453559 6514 raft_consensus.cc:729] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.462946 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.465476 6544 leader_election.cc:288] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.465937 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.466831 6471 raft_consensus.cc:2454] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:04:58.469971 6364 leader_election.cc:302] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
W20240503 14:04:58.471887 6362 leader_election.cc:341] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.475078 6544 raft_consensus.cc:2781] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.473615 6220 catalog_manager.cc:5484] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:58.476214 6544 raft_consensus.cc:482] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.476958 6544 raft_consensus.cc:3037] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.483511 6513 tablet_bootstrap.cc:654] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.485219 6512 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:58.485915 6512 ts_tablet_manager.cc:1367] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.183s user 0.004s sys 0.021s
I20240503 14:04:58.474639 6514 consensus_queue.cc:260] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.491499 6512 raft_consensus.cc:348] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.492455 6512 raft_consensus.cc:374] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.492833 6512 raft_consensus.cc:729] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.492933 6514 ts_tablet_manager.cc:1398] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.050s user 0.011s sys 0.001s
I20240503 14:04:58.494473 6514 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.501538 6544 raft_consensus.cc:504] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:04:58.503733 6364 consensus_peers.cc:485] T 1fd965edad81421d8b03f93895f80ebe P 90a66da3b59e4511bdc1d35930fe558d -> Peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Couldn't send request to peer 8e7d4e1e07e94f21b75ba42bf86334e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:58.504385 6514 tablet_bootstrap.cc:654] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.505492 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:04:58.507261 6362 leader_election.cc:341] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.508555 6512 consensus_queue.cc:260] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.515079 6512 ts_tablet_manager.cc:1398] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20240503 14:04:58.520764 6544 leader_election.cc:288] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.522460 6512 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:58.522852 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:58.523609 6471 raft_consensus.cc:3037] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.530359 6471 raft_consensus.cc:2454] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:04:58.531854 6364 leader_election.cc:302] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.533833 6545 raft_consensus.cc:2781] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.534629 6545 raft_consensus.cc:686] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:04:58.535710 6545 consensus_queue.cc:237] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.538024 6513 tablet_bootstrap.cc:492] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.538633 6513 ts_tablet_manager.cc:1367] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.013s
I20240503 14:04:58.540004 6514 tablet_bootstrap.cc:492] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.540652 6514 ts_tablet_manager.cc:1367] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.046s user 0.027s sys 0.012s
I20240503 14:04:58.540824 6568 raft_consensus.cc:482] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.541915 6512 tablet_bootstrap.cc:654] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.541386 6568 raft_consensus.cc:504] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.544898 6568 leader_election.cc:288] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.544523 6514 raft_consensus.cc:348] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.545634 6514 raft_consensus.cc:374] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.546077 6514 raft_consensus.cc:729] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.546391 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.544610 6513 raft_consensus.cc:348] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.547400 6513 raft_consensus.cc:374] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.547691 6513 raft_consensus.cc:729] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.547049 6514 consensus_queue.cc:260] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.548367 6513 consensus_queue.cc:260] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.550236 6514 ts_tablet_manager.cc:1398] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:04:58.550274 6513 ts_tablet_manager.cc:1398] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:04:58.551559 6513 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
W20240503 14:04:58.551613 6362 leader_election.cc:341] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.551780 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:58.553550 6364 leader_election.cc:341] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.554294 6514 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.558363 6513 tablet_bootstrap.cc:654] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.554069 6364 leader_election.cc:302] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.561471 6568 raft_consensus.cc:2726] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:58.564354 6512 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:58.564960 6512 ts_tablet_manager.cc:1367] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.007s
I20240503 14:04:58.571098 6514 tablet_bootstrap.cc:654] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.568495 6512 raft_consensus.cc:348] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.573324 6512 raft_consensus.cc:374] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.573737 6512 raft_consensus.cc:729] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.574630 6512 consensus_queue.cc:260] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.577006 6513 tablet_bootstrap.cc:492] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.578333 6513 ts_tablet_manager.cc:1367] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.007s
I20240503 14:04:58.581192 6513 raft_consensus.cc:348] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.582002 6513 raft_consensus.cc:374] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.582253 6512 ts_tablet_manager.cc:1398] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.017s user 0.006s sys 0.004s
I20240503 14:04:58.582341 6513 raft_consensus.cc:729] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.583345 6513 consensus_queue.cc:260] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.584580 6512 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:58.585917 6513 ts_tablet_manager.cc:1398] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:04:58.586024 6514 tablet_bootstrap.cc:492] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.586746 6514 ts_tablet_manager.cc:1367] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.014s
I20240503 14:04:58.587280 6513 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.589691 6514 raft_consensus.cc:348] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.590428 6514 raft_consensus.cc:374] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.590709 6514 raft_consensus.cc:729] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.591485 6514 consensus_queue.cc:260] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.593662 6514 ts_tablet_manager.cc:1398] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:04:58.594591 6514 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.595649 6513 tablet_bootstrap.cc:654] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.592514 6220 catalog_manager.cc:5484] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:58.604137 6514 tablet_bootstrap.cc:654] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.614701 6512 tablet_bootstrap.cc:654] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.617830 6554 raft_consensus.cc:482] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.618563 6554 raft_consensus.cc:504] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.620859 6514 tablet_bootstrap.cc:492] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.621464 6514 ts_tablet_manager.cc:1367] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.000s
I20240503 14:04:58.622639 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.623246 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.624167 6471 raft_consensus.cc:2454] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
W20240503 14:04:58.624441 6286 leader_election.cc:341] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.624243 6514 raft_consensus.cc:348] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.625296 6514 raft_consensus.cc:374] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.625644 6514 raft_consensus.cc:729] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.625881 6289 leader_election.cc:302] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.626464 6514 consensus_queue.cc:260] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.627103 6552 raft_consensus.cc:2781] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.627632 6552 raft_consensus.cc:482] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.628103 6552 raft_consensus.cc:3037] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.629027 6514 ts_tablet_manager.cc:1398] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:04:58.632594 6554 leader_election.cc:288] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.633466 6514 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.635512 6554 raft_consensus.cc:482] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.635733 6518 raft_consensus.cc:482] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.636667 6553 raft_consensus.cc:482] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.636512 6518 raft_consensus.cc:504] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.637249 6553 raft_consensus.cc:504] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.637948 6554 raft_consensus.cc:504] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.640311 6545 raft_consensus.cc:482] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.640027 6553 leader_election.cc:288] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.640902 6545 raft_consensus.cc:504] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.641672 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.642403 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.642624 6397 raft_consensus.cc:2454] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:58.643050 6554 leader_election.cc:288] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.643512 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.644405 6396 raft_consensus.cc:2454] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:58.646155 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.646981 6322 raft_consensus.cc:2454] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:58.648231 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:58.650904 6435 leader_election.cc:341] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.651786 6435 leader_election.cc:302] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:58.653026 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.653411 6545 leader_election.cc:288] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.654395 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:58.654793 6362 leader_election.cc:341] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:04:58.656103 6364 leader_election.cc:341] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.656795 6514 tablet_bootstrap.cc:654] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.656420 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.656711 6364 leader_election.cc:302] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.657091 6286 leader_election.cc:302] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:58.656440 6436 leader_election.cc:302] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.658121 6554 raft_consensus.cc:2781] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.657440 6472 raft_consensus.cc:2454] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:58.658540 6554 raft_consensus.cc:482] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.659404 6554 raft_consensus.cc:3037] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.659368 6551 raft_consensus.cc:2781] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.659940 6551 raft_consensus.cc:482] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.660209 6518 leader_election.cc:288] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.660432 6551 raft_consensus.cc:3037] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.659976 6553 raft_consensus.cc:2781] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.661460 6553 raft_consensus.cc:482] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.661870 6553 raft_consensus.cc:3037] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.658509 6545 raft_consensus.cc:2726] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:04:58.658499 6436 leader_election.cc:341] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.673292 6518 raft_consensus.cc:482] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.674096 6513 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.673900 6518 raft_consensus.cc:504] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.674569 6513 ts_tablet_manager.cc:1367] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.088s user 0.028s sys 0.032s
I20240503 14:04:58.675513 6512 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:58.676273 6512 ts_tablet_manager.cc:1367] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.011s
I20240503 14:04:58.677546 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.677407 6513 raft_consensus.cc:348] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.678299 6513 raft_consensus.cc:374] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.678606 6513 raft_consensus.cc:729] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.679387 6513 consensus_queue.cc:260] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.680316 6512 raft_consensus.cc:348] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.681133 6512 raft_consensus.cc:374] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.681330 6518 leader_election.cc:288] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.681540 6512 raft_consensus.cc:729] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.681067 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd965edad81421d8b03f93895f80ebe" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.682260 6513 ts_tablet_manager.cc:1398] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.007s user 0.007s sys 0.002s
I20240503 14:04:58.682482 6512 consensus_queue.cc:260] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.683240 6513 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.683763 6435 leader_election.cc:302] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.686007 6512 ts_tablet_manager.cc:1398] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 14:04:58.689343 6512 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:58.692230 6518 raft_consensus.cc:2726] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:58.694207 6551 raft_consensus.cc:504] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.704707 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:58.705855 6397 raft_consensus.cc:3037] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.707000 6553 raft_consensus.cc:504] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.709817 6553 leader_election.cc:288] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.710265 6512 tablet_bootstrap.cc:654] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.711242 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:04:58.713117 6435 leader_election.cc:341] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.713912 6551 leader_election.cc:288] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.706879 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:58.717060 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:58.717813 6322 raft_consensus.cc:3037] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.719238 6514 tablet_bootstrap.cc:492] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.719708 6514 ts_tablet_manager.cc:1367] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.087s user 0.022s sys 0.035s
I20240503 14:04:58.722546 6513 tablet_bootstrap.cc:654] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.715447 6552 raft_consensus.cc:504] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.719731 6554 raft_consensus.cc:504] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.725528 6396 raft_consensus.cc:3037] T d070812c38ea4aeda6ce94fac9ed7de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.727059 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:58.727813 6398 raft_consensus.cc:3037] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.727855 6514 raft_consensus.cc:348] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.728866 6514 raft_consensus.cc:374] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.729203 6514 raft_consensus.cc:729] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
W20240503 14:04:58.730402 6436 leader_election.cc:341] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.731083 6514 consensus_queue.cc:260] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.731828 6537 consensus_queue.cc:1035] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:58.733314 6514 ts_tablet_manager.cc:1398] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20240503 14:04:58.734273 6514 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.738888 6322 raft_consensus.cc:2454] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:58.740739 6514 tablet_bootstrap.cc:654] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.748045 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:04:58.751876 6286 leader_election.cc:341] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.741432 6554 leader_election.cc:288] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.745775 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca47ba300cb146968aba2dc3e5f4857a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:58.748442 6552 leader_election.cc:288] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.752741 6537 consensus_queue.cc:1035] T d070812c38ea4aeda6ce94fac9ed7de0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:58.753405 6471 raft_consensus.cc:3037] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.754772 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6845d794633f4f47a07ae27a88362ffa" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:58.759990 6470 raft_consensus.cc:3037] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.749009 6398 raft_consensus.cc:2454] T ca47ba300cb146968aba2dc3e5f4857a P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:04:58.765066 6286 leader_election.cc:302] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:58.765362 6397 raft_consensus.cc:2454] T 801af7dc07784c89890730a686553023 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:58.766191 6537 raft_consensus.cc:2781] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.766912 6435 leader_election.cc:302] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:58.767994 6551 raft_consensus.cc:2781] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.766830 6537 raft_consensus.cc:686] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:04:58.768640 6551 raft_consensus.cc:686] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:58.769289 6537 consensus_queue.cc:237] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.769685 6551 consensus_queue.cc:237] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.774951 6470 raft_consensus.cc:2454] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:04:58.777686 6436 leader_election.cc:302] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.788761 6551 raft_consensus.cc:2781] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.789366 6289 leader_election.cc:302] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:58.789494 6551 raft_consensus.cc:686] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:58.792815 6512 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:58.793330 6512 ts_tablet_manager.cc:1367] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.104s user 0.013s sys 0.003s
I20240503 14:04:58.792443 6551 consensus_queue.cc:237] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.795801 6395 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.775874 6471 raft_consensus.cc:2454] T ca47ba300cb146968aba2dc3e5f4857a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:04:58.781515 6514 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.797634 6514 ts_tablet_manager.cc:1367] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.064s user 0.017s sys 0.001s
I20240503 14:04:58.799238 6552 raft_consensus.cc:2781] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:58.800834 6514 raft_consensus.cc:348] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.801630 6514 raft_consensus.cc:374] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.801932 6514 raft_consensus.cc:729] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.802635 6514 consensus_queue.cc:260] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.806001 6514 ts_tablet_manager.cc:1398] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:04:58.807121 6514 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.815755 6514 tablet_bootstrap.cc:654] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.816833 6512 raft_consensus.cc:348] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.817934 6512 raft_consensus.cc:374] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.818291 6512 raft_consensus.cc:729] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.819227 6512 consensus_queue.cc:260] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.782430 6513 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:58.784205 6553 raft_consensus.cc:482] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.829694 6553 raft_consensus.cc:504] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.833823 6553 leader_election.cc:288] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.836530 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.837579 6398 raft_consensus.cc:2454] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:58.835904 6513 ts_tablet_manager.cc:1367] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.153s user 0.012s sys 0.017s
I20240503 14:04:58.839668 6435 leader_election.cc:302] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:58.842146 6513 raft_consensus.cc:348] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.843042 6513 raft_consensus.cc:374] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.843338 6513 raft_consensus.cc:729] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.846352 6553 raft_consensus.cc:2781] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.847374 6553 raft_consensus.cc:482] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.847826 6553 raft_consensus.cc:3037] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.849961 6545 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.850572 6545 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.853931 6568 raft_consensus.cc:482] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.854601 6568 raft_consensus.cc:504] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.854607 6518 consensus_queue.cc:1035] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20240503 14:04:58.865924 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:58.886039 6513 consensus_queue.cc:260] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.887241 6544 raft_consensus.cc:482] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.887825 6544 raft_consensus.cc:504] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.888507 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.889146 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.891188 6568 leader_election.cc:288] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.891491 6544 leader_election.cc:288] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:04:58.892522 6364 leader_election.cc:341] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:58.894047 6589 raft_consensus.cc:482] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.894536 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.894613 6589 raft_consensus.cc:504] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.897164 6545 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.897485 6588 raft_consensus.cc:482] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.902349 6512 ts_tablet_manager.cc:1398] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.109s user 0.012s sys 0.001s
I20240503 14:04:58.903527 6512 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:58.911037 6513 ts_tablet_manager.cc:1398] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.072s user 0.005s sys 0.002s
I20240503 14:04:58.900110 6552 raft_consensus.cc:686] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:04:58.922529 6552 consensus_queue.cc:237] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.924294 6545 raft_consensus.cc:482] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:58.924916 6545 raft_consensus.cc:504] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.928443 6513 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:58.917332 6589 leader_election.cc:288] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:04:58.901185 6286 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 104493us
W20240503 14:04:58.909293 6289 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 111300us
I20240503 14:04:58.933351 6545 leader_election.cc:288] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:58.935117 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.936010 6512 tablet_bootstrap.cc:654] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:58.937629 6514 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:58.940346 6514 ts_tablet_manager.cc:1367] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.133s user 0.029s sys 0.047s
I20240503 14:04:58.915195 6553 raft_consensus.cc:504] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.946468 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.946918 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:58.947260 6323 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:04:58.947707 6395 raft_consensus.cc:2454] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:58.940694 6472 raft_consensus.cc:2454] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:04:58.898198 6588 raft_consensus.cc:504] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.942497 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.936901 6220 catalog_manager.cc:5484] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:58.948064 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:58.963865 6514 raft_consensus.cc:348] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.964911 6514 raft_consensus.cc:374] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:58.965260 6514 raft_consensus.cc:729] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:58.945627 6553 leader_election.cc:288] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:58.970454 6514 consensus_queue.cc:260] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:58.971612 6364 leader_election.cc:302] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:58.974642 6545 raft_consensus.cc:2781] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:58.975142 6545 raft_consensus.cc:482] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:58.975553 6545 raft_consensus.cc:3037] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.960232 6469 raft_consensus.cc:3037] T 1fd965edad81421d8b03f93895f80ebe P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.979455 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:58.983224 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.960163 6470 raft_consensus.cc:2454] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:58.985482 6319 raft_consensus.cc:2454] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:04:58.970064 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:58.989617 6553 consensus_queue.cc:1035] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:04:58.976862 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:58.969151 6286 leader_election.cc:302] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:58.982086 6322 raft_consensus.cc:2454] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:58.989344 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861ddaea971348a596840c74fb1e8a71" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.000885 6514 ts_tablet_manager.cc:1398] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.060s user 0.005s sys 0.008s
I20240503 14:04:59.003602 6589 raft_consensus.cc:2781] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.004559 6589 raft_consensus.cc:482] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.005062 6589 raft_consensus.cc:3037] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.005184 6514 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:58.997311 6395 raft_consensus.cc:3037] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:58.996392 6323 raft_consensus.cc:2454] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:04:59.010146 6362 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:04:59.011960 6544 raft_consensus.cc:2781] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:04:59.004712 6320 raft_consensus.cc:2379] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:04:59.012375 6544 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8e7d4e1e07e94f21b75ba42bf86334e1)
I20240503 14:04:58.975917 6219 catalog_manager.cc:5484] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:59.013561 6544 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:04:59.008824 6286 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 47015.3us
W20240503 14:04:59.020062 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08000c06e0 after lost signal to thread 5102
I20240503 14:04:59.032661 6219 catalog_manager.cc:5484] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:59.038081 6589 raft_consensus.cc:504] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.041903 6544 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.044929 6588 leader_election.cc:288] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.045514 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.064509 6589 leader_election.cc:288] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.065438 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:59.065727 6544 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.093196 6598 raft_consensus.cc:482] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.096645 6395 raft_consensus.cc:2454] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:59.100680 6545 raft_consensus.cc:504] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.111410 6397 raft_consensus.cc:2379] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 90a66da3b59e4511bdc1d35930fe558d in this term.
I20240503 14:04:59.109654 6568 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20240503 14:04:59.107517 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 66372.8us
I20240503 14:04:59.115764 6545 leader_election.cc:288] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.114030 6435 leader_election.cc:302] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.116394 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
W20240503 14:04:59.115837 6503 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.4.250.254:36245, user_credentials={real_user=slave}} blocked reactor thread for 85728us
I20240503 14:04:59.121845 6513 tablet_bootstrap.cc:654] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:59.118556 6493 debug-util.cc:398] Leaking SignalData structure 0x7b0800042ae0 after lost signal to thread 6255
I20240503 14:04:59.123330 6518 raft_consensus.cc:2781] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.123132 6362 leader_election.cc:302] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:04:59.131093 6518 raft_consensus.cc:686] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:59.132186 6362 leader_election.cc:302] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.132299 6518 consensus_queue.cc:237] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.134045 6568 raft_consensus.cc:2781] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.134527 6568 raft_consensus.cc:482] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.134971 6568 raft_consensus.cc:3037] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.135716 6545 raft_consensus.cc:3037] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.137492 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102ad45eeee4cb4a3d5b7f5656a70d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:04:59.139304 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08001594c0 after lost signal to thread 6347
W20240503 14:04:59.140924 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0c80 after lost signal to thread 6422
I20240503 14:04:59.147190 6514 tablet_bootstrap.cc:654] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.108937 6598 raft_consensus.cc:504] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.156244 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:59.157171 6289 leader_election.cc:302] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:59.159662 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.160743 6323 raft_consensus.cc:2379] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:04:59.161722 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc690ef263249609cf947bfcd0d4170" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:59.163300 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.167945 6289 leader_election.cc:302] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:59.166055 6589 raft_consensus.cc:2726] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:59.168212 6598 leader_election.cc:288] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.157227 6471 raft_consensus.cc:3037] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.170763 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.171977 6512 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:59.173123 6512 ts_tablet_manager.cc:1367] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.270s user 0.019s sys 0.010s
I20240503 14:04:59.171656 6589 raft_consensus.cc:2726] T cbc690ef263249609cf947bfcd0d4170 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240503 14:04:59.169533 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08000ec780 after lost signal to thread 6496
I20240503 14:04:59.168726 6362 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.174160 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.174597 6544 raft_consensus.cc:2726] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240503 14:04:59.175805 6436 leader_election.cc:341] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:04:59.174526 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08000f8d40 after lost signal to thread 6527
W20240503 14:04:59.177035 6493 debug-util.cc:398] Leaking SignalData structure 0x7b08001f0720 after lost signal to thread 6536
I20240503 14:04:59.164038 6568 raft_consensus.cc:504] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:04:59.176460 6435 leader_election.cc:341] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.181069 6435 leader_election.cc:302] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.181864 6545 raft_consensus.cc:2726] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:59.182128 6598 raft_consensus.cc:2726] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:59.179621 6512 raft_consensus.cc:348] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.183223 6512 raft_consensus.cc:374] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.183514 6512 raft_consensus.cc:729] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.184072 6512 consensus_queue.cc:260] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.187649 6512 ts_tablet_manager.cc:1398] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20240503 14:04:59.188294 6322 raft_consensus.cc:3037] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.188844 6512 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:59.188580 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.189348 6323 raft_consensus.cc:3037] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.189561 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
W20240503 14:04:59.191313 6364 leader_election.cc:341] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.193984 6568 leader_election.cc:288] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.199126 6589 consensus_queue.cc:1035] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.205073 6513 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:59.205665 6513 ts_tablet_manager.cc:1367] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.278s user 0.021s sys 0.013s
I20240503 14:04:59.206317 6514 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.206805 6514 ts_tablet_manager.cc:1367] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.202s user 0.005s sys 0.011s
I20240503 14:04:59.208011 6471 raft_consensus.cc:2454] T cbc690ef263249609cf947bfcd0d4170 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:04:59.208535 6513 raft_consensus.cc:348] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.209353 6513 raft_consensus.cc:374] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.203666 6323 raft_consensus.cc:2454] T ce2b1e76121d4515956175ba6bbe8fdb P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:04:59.209633 6513 raft_consensus.cc:729] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.209575 6364 leader_election.cc:302] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.210547 6513 consensus_queue.cc:260] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.210882 6568 raft_consensus.cc:2781] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.213433 6568 raft_consensus.cc:686] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:04:59.214138 6513 ts_tablet_manager.cc:1398] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20240503 14:04:59.214550 6568 consensus_queue.cc:237] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.212430 6362 leader_election.cc:302] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:04:59.216845 6544 raft_consensus.cc:2781] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.217514 6544 raft_consensus.cc:686] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:04:59.218724 6544 consensus_queue.cc:237] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.215282 6513 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:59.226536 6514 raft_consensus.cc:348] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.227437 6514 raft_consensus.cc:374] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.227732 6514 raft_consensus.cc:729] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.228458 6514 consensus_queue.cc:260] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.231786 6514 ts_tablet_manager.cc:1398] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20240503 14:04:59.232721 6514 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.253860 6513 tablet_bootstrap.cc:654] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.253211 6589 raft_consensus.cc:482] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.263077 6545 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.258103 6589 raft_consensus.cc:504] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.276136 6598 raft_consensus.cc:482] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.271852 6587 consensus_queue.cc:1035] T ca47ba300cb146968aba2dc3e5f4857a P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240503 14:04:59.278060 6589 leader_election.cc:288] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.279242 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.276862 6598 raft_consensus.cc:504] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.284713 6514 tablet_bootstrap.cc:654] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.285162 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388b249118b41faa4cb6efffb8fecea" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:59.285465 6598 leader_election.cc:288] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.288004 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.288964 6289 leader_election.cc:302] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:59.291395 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:59.293452 6435 leader_election.cc:341] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.297693 6512 tablet_bootstrap.cc:654] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
W20240503 14:04:59.299278 6436 leader_election.cc:341] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.302206 6587 raft_consensus.cc:2726] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:04:59.302613 6436 leader_election.cc:302] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.303812 6598 raft_consensus.cc:2726] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:59.303565 6219 catalog_manager.cc:5484] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:59.318301 6513 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:59.318854 6513 ts_tablet_manager.cc:1367] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.008s
I20240503 14:04:59.324405 6513 raft_consensus.cc:348] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.324970 6513 raft_consensus.cc:374] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.325201 6513 raft_consensus.cc:729] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.326164 6598 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.328616 6513 consensus_queue.cc:260] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.332299 6514 tablet_bootstrap.cc:492] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.332806 6514 ts_tablet_manager.cc:1367] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.008s
I20240503 14:04:59.335747 6514 raft_consensus.cc:348] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.336561 6514 raft_consensus.cc:374] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.336830 6514 raft_consensus.cc:729] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.336951 6598 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.338681 6512 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:59.339217 6512 ts_tablet_manager.cc:1367] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.151s user 0.010s sys 0.006s
I20240503 14:04:59.342376 6512 raft_consensus.cc:348] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.343139 6512 raft_consensus.cc:374] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.343436 6512 raft_consensus.cc:729] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.343616 6220 catalog_manager.cc:5484] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:59.345767 6513 ts_tablet_manager.cc:1398] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
I20240503 14:04:59.346027 6518 consensus_queue.cc:1035] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.346815 6513 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:59.346361 6220 catalog_manager.cc:5484] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:59.349593 6512 consensus_queue.cc:260] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.353396 6512 ts_tablet_manager.cc:1398] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.014s user 0.004s sys 0.005s
I20240503 14:04:59.367411 6589 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.367766 6514 consensus_queue.cc:260] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.371805 6513 tablet_bootstrap.cc:654] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.372424 6514 ts_tablet_manager.cc:1398] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.039s user 0.008s sys 0.001s
I20240503 14:04:59.375734 6514 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.376572 6395 raft_consensus.cc:3037] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.380625 6512 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:59.410153 6514 tablet_bootstrap.cc:654] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.422693 6587 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.430977 6568 raft_consensus.cc:482] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.431327 6396 raft_consensus.cc:1226] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Rejecting Update request from peer 8e7d4e1e07e94f21b75ba42bf86334e1 for earlier term 1. Current term is 2. Ops: []
I20240503 14:04:59.431599 6568 raft_consensus.cc:504] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:04:59.437795 6436 consensus_peers.cc:485] T 801af7dc07784c89890730a686553023 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:04:59.434963 6553 consensus_queue.cc:1046] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240503 14:04:59.438704 6568 leader_election.cc:288] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.440166 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:59.441659 6512 tablet_bootstrap.cc:654] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.442438 6598 raft_consensus.cc:3032] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:04:59.443377 6598 raft_consensus.cc:729] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:59.444218 6598 consensus_queue.cc:260] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.445241 6514 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.445488 6598 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:04:59.445720 6514 ts_tablet_manager.cc:1367] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.005s
W20240503 14:04:59.447307 6364 leader_election.cc:341] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.449317 6514 raft_consensus.cc:348] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.450084 6514 raft_consensus.cc:374] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.450366 6514 raft_consensus.cc:729] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.451115 6514 consensus_queue.cc:260] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.453863 6514 ts_tablet_manager.cc:1398] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:04:59.454892 6514 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.457337 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:59.463953 6362 leader_election.cc:341] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.464615 6362 leader_election.cc:302] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.467643 6544 raft_consensus.cc:2726] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:04:59.470695 6514 tablet_bootstrap.cc:654] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.472736 6513 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:59.473134 6513 ts_tablet_manager.cc:1367] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.127s user 0.015s sys 0.006s
I20240503 14:04:59.480198 6513 raft_consensus.cc:348] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.480939 6513 raft_consensus.cc:374] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.481227 6513 raft_consensus.cc:729] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.482825 6513 consensus_queue.cc:260] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.485575 6513 ts_tablet_manager.cc:1398] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20240503 14:04:59.484915 6218 catalog_manager.cc:5484] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:59.486698 6513 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:59.494612 6513 tablet_bootstrap.cc:654] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.504901 6512 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:59.505432 6512 ts_tablet_manager.cc:1367] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.125s user 0.009s sys 0.007s
I20240503 14:04:59.507019 6514 tablet_bootstrap.cc:492] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.507602 6514 ts_tablet_manager.cc:1367] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.008s
I20240503 14:04:59.508581 6512 raft_consensus.cc:348] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.509508 6512 raft_consensus.cc:374] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.509799 6512 raft_consensus.cc:729] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.510485 6512 consensus_queue.cc:260] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.510970 6514 raft_consensus.cc:348] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.511778 6514 raft_consensus.cc:374] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.512058 6514 raft_consensus.cc:729] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.513080 6512 ts_tablet_manager.cc:1398] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240503 14:04:59.512725 6514 consensus_queue.cc:260] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.514066 6512 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:59.515324 6514 ts_tablet_manager.cc:1398] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240503 14:04:59.516479 6514 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.519042 6323 raft_consensus.cc:3037] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.521042 6512 tablet_bootstrap.cc:654] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.523458 6514 tablet_bootstrap.cc:654] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.538367 6514 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.538905 6514 ts_tablet_manager.cc:1367] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.013s
I20240503 14:04:59.541868 6514 raft_consensus.cc:348] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.542649 6514 raft_consensus.cc:374] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.542951 6514 raft_consensus.cc:729] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.543639 6514 consensus_queue.cc:260] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.546866 6514 ts_tablet_manager.cc:1398] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 14:04:59.547868 6514 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.555855 6514 tablet_bootstrap.cc:654] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.558734 6596 raft_consensus.cc:482] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.559406 6596 raft_consensus.cc:504] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.566162 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ee8e66a5b74a4394a4e339a0af957d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.567041 6395 raft_consensus.cc:2454] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:59.569495 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ee8e66a5b74a4394a4e339a0af957d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.572024 6435 leader_election.cc:302] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:59.573501 6598 raft_consensus.cc:2781] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:04:59.573398 6436 leader_election.cc:341] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.573697 6596 leader_election.cc:288] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.574215 6598 raft_consensus.cc:482] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.575178 6518 consensus_queue.cc:1035] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.575572 6598 raft_consensus.cc:3037] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.595083 6514 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.595660 6514 ts_tablet_manager.cc:1367] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.006s
I20240503 14:04:59.598814 6514 raft_consensus.cc:348] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.599607 6514 raft_consensus.cc:374] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.599870 6514 raft_consensus.cc:729] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.600658 6514 consensus_queue.cc:260] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.603147 6514 ts_tablet_manager.cc:1398] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240503 14:04:59.604115 6514 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.611124 6514 tablet_bootstrap.cc:654] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.616112 6512 tablet_bootstrap.cc:492] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:59.616703 6512 ts_tablet_manager.cc:1367] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.103s user 0.028s sys 0.021s
I20240503 14:04:59.618533 6598 raft_consensus.cc:504] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.619825 6512 raft_consensus.cc:348] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.620651 6512 raft_consensus.cc:374] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.620922 6512 raft_consensus.cc:729] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.622661 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ee8e66a5b74a4394a4e339a0af957d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:59.623467 6395 raft_consensus.cc:3037] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.626693 6512 consensus_queue.cc:260] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.627583 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ee8e66a5b74a4394a4e339a0af957d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.628715 6512 ts_tablet_manager.cc:1398] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:04:59.629514 6512 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:59.630013 6596 raft_consensus.cc:482] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.630613 6596 raft_consensus.cc:504] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.632977 6513 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:59.633589 6513 ts_tablet_manager.cc:1367] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.147s user 0.020s sys 0.042s
I20240503 14:04:59.628629 6598 leader_election.cc:288] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.638458 6596 leader_election.cc:288] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.628417 6322 raft_consensus.cc:3037] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.640925 6518 raft_consensus.cc:482] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.642197 6513 raft_consensus.cc:348] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.644814 6513 raft_consensus.cc:374] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.645114 6513 raft_consensus.cc:729] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.645992 6552 raft_consensus.cc:482] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.645969 6513 consensus_queue.cc:260] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.646693 6552 raft_consensus.cc:504] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.649103 6513 ts_tablet_manager.cc:1398] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.015s user 0.005s sys 0.004s
I20240503 14:04:59.648886 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.649902 6398 raft_consensus.cc:2454] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:59.650234 6513 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:59.651306 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.651595 6518 raft_consensus.cc:504] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.652294 6323 raft_consensus.cc:2454] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:59.651201 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.654266 6436 leader_election.cc:302] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:04:59.656863 6518 leader_election.cc:288] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.658718 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:04:59.660586 6286 leader_election.cc:341] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.660315 6397 raft_consensus.cc:2454] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:59.663388 6553 raft_consensus.cc:2781] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.663933 6553 raft_consensus.cc:482] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.664440 6553 raft_consensus.cc:3037] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.660454 6395 raft_consensus.cc:2454] T c4ee8e66a5b74a4394a4e339a0af957d P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:59.668816 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.670300 6513 tablet_bootstrap.cc:654] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.666203 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:04:59.671079 6469 raft_consensus.cc:2454] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:04:59.662448 6435 leader_election.cc:302] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:59.675714 6551 raft_consensus.cc:2781] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.676285 6551 raft_consensus.cc:482] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.676757 6551 raft_consensus.cc:3037] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.676481 6435 leader_election.cc:302] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
W20240503 14:04:59.677945 6436 leader_election.cc:341] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.680577 6289 leader_election.cc:302] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:04:59.681836 6589 raft_consensus.cc:2781] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.664919 6552 leader_election.cc:288] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.682343 6589 raft_consensus.cc:482] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:04:59.678843 6596 raft_consensus.cc:2781] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.684065 6518 consensus_queue.cc:1035] T 7102ad45eeee4cb4a3d5b7f5656a70d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.682817 6589 raft_consensus.cc:3037] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.697124 6553 raft_consensus.cc:504] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.698326 6589 raft_consensus.cc:504] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.699083 6544 consensus_queue.cc:1035] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.700976 6553 leader_election.cc:288] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.701493 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.703316 6320 raft_consensus.cc:3037] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.703464 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:04:59.706280 6469 raft_consensus.cc:3037] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.703672 6589 leader_election.cc:288] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:04:59.704469 6596 raft_consensus.cc:686] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:59.727074 6322 raft_consensus.cc:2454] T c4ee8e66a5b74a4394a4e339a0af957d P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:59.728614 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f379675884943d6b074d7af75fdbf64" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:59.730180 6398 raft_consensus.cc:3037] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.728713 6596 consensus_queue.cc:237] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.735719 6514 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:04:59.736609 6514 ts_tablet_manager.cc:1367] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.037s
I20240503 14:04:59.738891 6469 raft_consensus.cc:2454] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:04:59.740156 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:59.745321 6289 leader_election.cc:302] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
W20240503 14:04:59.746482 6286 leader_election.cc:341] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.748044 6514 raft_consensus.cc:348] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.750707 6514 raft_consensus.cc:374] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.751016 6514 raft_consensus.cc:729] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.751602 6544 consensus_queue.cc:1035] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.748445 6469 raft_consensus.cc:3037] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.751734 6514 consensus_queue.cc:260] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.766546 6512 tablet_bootstrap.cc:654] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.769662 6514 ts_tablet_manager.cc:1398] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:04:59.771196 6551 raft_consensus.cc:504] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.773029 6514 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:04:59.754477 6589 raft_consensus.cc:2781] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.777258 6398 raft_consensus.cc:2454] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:59.784219 6435 leader_election.cc:302] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:59.788154 6632 raft_consensus.cc:2781] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.788846 6632 raft_consensus.cc:686] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:59.780448 6589 raft_consensus.cc:686] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:04:59.790635 6632 consensus_queue.cc:237] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.791525 6589 consensus_queue.cc:237] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.802742 6553 raft_consensus.cc:482] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.803793 6553 raft_consensus.cc:504] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.807284 6513 tablet_bootstrap.cc:492] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:04:59.848436 6513 ts_tablet_manager.cc:1367] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.198s user 0.043s sys 0.031s
I20240503 14:04:59.847000 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:04:59.851408 6397 raft_consensus.cc:3037] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.846542 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:04:59.861886 6320 raft_consensus.cc:2454] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
W20240503 14:04:59.866683 6436 leader_election.cc:341] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.871006 6514 tablet_bootstrap.cc:654] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.832016 6551 leader_election.cc:288] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.836503 6544 consensus_queue.cc:1035] T ce2b1e76121d4515956175ba6bbe8fdb P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20240503 14:04:59.898732 6544 consensus_queue.cc:1035] T cbc690ef263249609cf947bfcd0d4170 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:04:59.904413 6551 raft_consensus.cc:482] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:04:59.918767 6551 raft_consensus.cc:504] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.925192 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:04:59.911309 6362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 48267.8us
W20240503 14:04:59.905972 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 50498.9us
I20240503 14:04:59.918516 6553 leader_election.cc:288] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.930621 6397 raft_consensus.cc:2454] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:04:59.934294 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.935212 6395 raft_consensus.cc:2454] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:04:59.928817 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce2b1e76121d4515956175ba6bbe8fdb" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:04:59.935526 6219 catalog_manager.cc:5484] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:04:59.938759 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:04:59.934290 6435 leader_election.cc:302] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.929379 6551 leader_election.cc:288] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:04:59.940862 6435 leader_election.cc:302] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:04:59.941537 6551 raft_consensus.cc:2781] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:04:59.941931 6632 raft_consensus.cc:2781] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:04:59.941021 6220 catalog_manager.cc:5484] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:04:59.942394 6551 raft_consensus.cc:686] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:04:59.942451 6632 raft_consensus.cc:482] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:04:59.942839 6436 leader_election.cc:341] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:04:59.943189 6632 raft_consensus.cc:3037] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:04:59.944932 6513 raft_consensus.cc:348] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.945533 6513 raft_consensus.cc:374] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.945744 6513 raft_consensus.cc:729] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.942260 6435 leader_election.cc:302] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:04:59.947643 6513 consensus_queue.cc:260] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.948375 6512 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:04:59.949083 6512 ts_tablet_manager.cc:1367] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.320s user 0.022s sys 0.004s
I20240503 14:04:59.950955 6513 ts_tablet_manager.cc:1398] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.102s user 0.007s sys 0.000s
I20240503 14:04:59.951992 6553 raft_consensus.cc:2726] T ce2b1e76121d4515956175ba6bbe8fdb P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:04:59.952020 6513 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:04:59.952129 6512 raft_consensus.cc:348] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:04:59.952298 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:04:59.952955 6512 raft_consensus.cc:374] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:04:59.955267 6512 raft_consensus.cc:729] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:04:59.956441 6512 consensus_queue.cc:260] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.959239 6512 ts_tablet_manager.cc:1398] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:04:59.960295 6512 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:04:59.971109 6513 tablet_bootstrap.cc:654] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:04:59.958909 6551 consensus_queue.cc:237] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:04:59.987926 6512 tablet_bootstrap.cc:654] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.001381 6514 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.001905 6514 ts_tablet_manager.cc:1367] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.229s user 0.015s sys 0.015s
I20240503 14:05:00.002909 6632 raft_consensus.cc:504] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.005728 6514 raft_consensus.cc:348] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.006762 6514 raft_consensus.cc:374] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.007118 6514 raft_consensus.cc:729] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.007956 6514 consensus_queue.cc:260] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.012506 6632 leader_election.cc:288] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.015293 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:00.016263 6398 raft_consensus.cc:3037] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.019615 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:00.025972 6436 leader_election.cc:341] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.027333 6514 ts_tablet_manager.cc:1398] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20240503 14:05:00.028216 6514 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.039053 6398 raft_consensus.cc:2454] T a4080dad66944ea291f6d69736cd2c23 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:00.039767 6513 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.042191 6513 ts_tablet_manager.cc:1367] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.012s
I20240503 14:05:00.043157 6435 leader_election.cc:302] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.045361 6513 raft_consensus.cc:348] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.046212 6513 raft_consensus.cc:374] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.046505 6513 raft_consensus.cc:729] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.047338 6513 consensus_queue.cc:260] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.048677 6551 raft_consensus.cc:2781] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.049741 6551 raft_consensus.cc:686] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:00.050100 6513 ts_tablet_manager.cc:1398] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:05:00.050876 6514 tablet_bootstrap.cc:654] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.051234 6513 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:00.051146 6551 consensus_queue.cc:237] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.054553 6512 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.055236 6512 ts_tablet_manager.cc:1367] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.012s
I20240503 14:05:00.058810 6513 tablet_bootstrap.cc:654] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.060133 6512 raft_consensus.cc:348] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.061043 6512 raft_consensus.cc:374] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.061461 6512 raft_consensus.cc:729] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.061817 6219 catalog_manager.cc:5484] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:00.062319 6512 consensus_queue.cc:260] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.064623 6219 catalog_manager.cc:5484] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:00.070631 6512 ts_tablet_manager.cc:1398] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.015s user 0.001s sys 0.006s
I20240503 14:05:00.072213 6512 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.091398 6512 tablet_bootstrap.cc:654] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.109958 6513 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.110641 6514 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.110769 6513 ts_tablet_manager.cc:1367] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.013s
I20240503 14:05:00.112221 6551 raft_consensus.cc:482] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.111231 6514 ts_tablet_manager.cc:1367] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.083s user 0.013s sys 0.000s
I20240503 14:05:00.115366 6513 raft_consensus.cc:348] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.116323 6513 raft_consensus.cc:374] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.116663 6513 raft_consensus.cc:729] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.118029 6513 consensus_queue.cc:260] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.112864 6551 raft_consensus.cc:504] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.122637 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e49c03a6cf8400b98ce8d56e55089e1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.123406 6551 leader_election.cc:288] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.125675 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e49c03a6cf8400b98ce8d56e55089e1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.126633 6398 raft_consensus.cc:2454] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:00.128522 6435 leader_election.cc:302] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:00.129660 6551 raft_consensus.cc:2781] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.130344 6551 raft_consensus.cc:482] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.130968 6551 raft_consensus.cc:3037] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.129982 6218 catalog_manager.cc:5484] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:00.132619 6513 ts_tablet_manager.cc:1398] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.021s user 0.004s sys 0.004s
I20240503 14:05:00.133713 6513 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
W20240503 14:05:00.135704 6436 leader_election.cc:341] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.137244 6514 raft_consensus.cc:348] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.138074 6514 raft_consensus.cc:374] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.138378 6514 raft_consensus.cc:729] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.139065 6514 consensus_queue.cc:260] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.141767 6514 ts_tablet_manager.cc:1398] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.028s user 0.005s sys 0.002s
I20240503 14:05:00.142614 6514 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.144991 6551 raft_consensus.cc:504] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.149662 6514 tablet_bootstrap.cc:654] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.150462 6551 leader_election.cc:288] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.147524 6513 tablet_bootstrap.cc:654] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.153244 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e49c03a6cf8400b98ce8d56e55089e1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:00.151959 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e49c03a6cf8400b98ce8d56e55089e1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:00.156546 6436 leader_election.cc:341] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.154137 6398 raft_consensus.cc:3037] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.171281 6512 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.171921 6512 ts_tablet_manager.cc:1367] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.028s
I20240503 14:05:00.176749 6512 raft_consensus.cc:348] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.177840 6512 raft_consensus.cc:374] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.178160 6512 raft_consensus.cc:729] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.178830 6512 consensus_queue.cc:260] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.183640 6513 tablet_bootstrap.cc:492] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.184244 6513 ts_tablet_manager.cc:1367] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.027s
I20240503 14:05:00.186643 6398 raft_consensus.cc:2454] T 0e49c03a6cf8400b98ce8d56e55089e1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:00.187525 6513 raft_consensus.cc:348] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.188185 6514 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.188364 6513 raft_consensus.cc:374] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.188733 6514 ts_tablet_manager.cc:1367] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.002s
I20240503 14:05:00.188387 6435 leader_election.cc:302] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.188794 6513 raft_consensus.cc:729] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.189620 6551 raft_consensus.cc:2781] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.181190 6512 ts_tablet_manager.cc:1398] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:05:00.190243 6551 raft_consensus.cc:686] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:00.191251 6512 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.191462 6514 raft_consensus.cc:348] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.192221 6514 raft_consensus.cc:374] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.192517 6514 raft_consensus.cc:729] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.191387 6551 consensus_queue.cc:237] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.193493 6514 consensus_queue.cc:260] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.196005 6514 ts_tablet_manager.cc:1398] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:05:00.197204 6513 consensus_queue.cc:260] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.199097 6514 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.201333 6513 ts_tablet_manager.cc:1398] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 14:05:00.204363 6513 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:00.206350 6514 tablet_bootstrap.cc:654] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.210076 6512 tablet_bootstrap.cc:654] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.212051 6552 raft_consensus.cc:482] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.212658 6552 raft_consensus.cc:504] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.214809 6552 leader_election.cc:288] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.216774 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.218313 6544 raft_consensus.cc:482] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.218672 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56476ec0a3874a83bc22ec0c101088c8" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.218940 6544 raft_consensus.cc:504] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.219758 6398 raft_consensus.cc:2379] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:00.222419 6544 leader_election.cc:288] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.222714 6286 leader_election.cc:302] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:00.223229 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.224141 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0fcc21be834ead9ab016a33b25928c" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.225152 6470 raft_consensus.cc:2379] T 6e0fcc21be834ead9ab016a33b25928c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:00.225934 6552 raft_consensus.cc:3037] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.227699 6513 tablet_bootstrap.cc:654] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.227624 6362 leader_election.cc:302] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.233938 6544 raft_consensus.cc:3037] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.242556 6551 raft_consensus.cc:482] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.243338 6551 raft_consensus.cc:504] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.244351 6218 catalog_manager.cc:5484] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:00.250794 6512 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.251442 6512 ts_tablet_manager.cc:1367] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.006s
I20240503 14:05:00.252354 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:00.254069 6436 leader_election.cc:341] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.258653 6551 leader_election.cc:288] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.261582 6552 raft_consensus.cc:2726] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:00.263144 6512 raft_consensus.cc:348] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.264199 6512 raft_consensus.cc:374] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.264755 6512 raft_consensus.cc:729] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.264159 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:00.267903 6435 leader_election.cc:341] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.268527 6435 leader_election.cc:302] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.270067 6551 raft_consensus.cc:2726] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:00.271260 6512 consensus_queue.cc:260] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.264117 6544 raft_consensus.cc:2726] T 6e0fcc21be834ead9ab016a33b25928c P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:00.273753 6512 ts_tablet_manager.cc:1398] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20240503 14:05:00.278815 6512 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.278854 6513 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.279489 6513 ts_tablet_manager.cc:1367] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.009s
I20240503 14:05:00.283519 6513 raft_consensus.cc:348] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.284379 6513 raft_consensus.cc:374] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.284718 6513 raft_consensus.cc:729] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.286590 6512 tablet_bootstrap.cc:654] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.285394 6513 consensus_queue.cc:260] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.288178 6514 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.288673 6514 ts_tablet_manager.cc:1367] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.022s
I20240503 14:05:00.291314 6514 raft_consensus.cc:348] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.293783 6513 ts_tablet_manager.cc:1398] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20240503 14:05:00.294201 6514 raft_consensus.cc:374] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.294545 6514 raft_consensus.cc:729] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.295662 6513 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:00.297194 6514 consensus_queue.cc:260] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.306522 6553 consensus_queue.cc:1035] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.314440 6632 consensus_queue.cc:1035] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.328711 6513 tablet_bootstrap.cc:654] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.327950 6514 ts_tablet_manager.cc:1398] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.039s user 0.016s sys 0.007s
I20240503 14:05:00.330067 6514 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.349447 6552 consensus_queue.cc:1035] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.351617 6553 raft_consensus.cc:482] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.351974 6596 consensus_queue.cc:1035] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.352291 6553 raft_consensus.cc:504] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.354779 6553 leader_election.cc:288] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.357306 6544 raft_consensus.cc:482] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.357918 6544 raft_consensus.cc:504] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.358171 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6985cc0483c34fcd95b3e56b8b59bfb4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.360780 6320 raft_consensus.cc:2454] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:00.363662 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.366513 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.367261 6470 raft_consensus.cc:2454] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:00.364743 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6985cc0483c34fcd95b3e56b8b59bfb4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:00.368633 6362 leader_election.cc:341] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.362074 6544 leader_election.cc:288] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.369580 6364 leader_election.cc:302] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.370445 6436 leader_election.cc:302] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
W20240503 14:05:00.371133 6435 leader_election.cc:341] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.371412 6553 raft_consensus.cc:2781] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.372000 6553 raft_consensus.cc:482] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.372279 6544 raft_consensus.cc:2781] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.372478 6553 raft_consensus.cc:3037] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.372814 6544 raft_consensus.cc:482] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.373277 6544 raft_consensus.cc:3037] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.375873 6514 tablet_bootstrap.cc:654] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.395421 6551 consensus_queue.cc:1035] T c4ee8e66a5b74a4394a4e339a0af957d P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.411314 6553 raft_consensus.cc:504] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.426725 6512 tablet_bootstrap.cc:492] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.427279 6512 ts_tablet_manager.cc:1367] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.010s
I20240503 14:05:00.429351 6513 tablet_bootstrap.cc:492] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.429997 6513 ts_tablet_manager.cc:1367] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.010s
I20240503 14:05:00.439020 6553 leader_election.cc:288] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.437309 6512 raft_consensus.cc:348] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.440325 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6985cc0483c34fcd95b3e56b8b59bfb4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:00.441076 6512 raft_consensus.cc:374] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.441464 6512 raft_consensus.cc:729] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.442307 6512 consensus_queue.cc:260] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.448153 6512 ts_tablet_manager.cc:1398] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.021s user 0.005s sys 0.004s
I20240503 14:05:00.450676 6552 raft_consensus.cc:482] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.454712 6512 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.455119 6552 raft_consensus.cc:504] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.450157 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6985cc0483c34fcd95b3e56b8b59bfb4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:00.461040 6322 raft_consensus.cc:3037] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.464404 6552 leader_election.cc:288] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.472841 6512 tablet_bootstrap.cc:654] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.476043 6513 raft_consensus.cc:348] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.476397 6552 consensus_queue.cc:1035] T 6e0fcc21be834ead9ab016a33b25928c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.476915 6513 raft_consensus.cc:374] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.479460 6513 raft_consensus.cc:729] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.480242 6513 consensus_queue.cc:260] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.486395 6396 raft_consensus.cc:3037] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.486698 6513 ts_tablet_manager.cc:1398] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.056s user 0.008s sys 0.000s
W20240503 14:05:00.482291 6664 log.cc:924] Time spent T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20240503 14:05:00.491261 6513 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:00.491969 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.492839 6397 raft_consensus.cc:2454] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:00.497906 6668 raft_consensus.cc:482] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.497959 6551 raft_consensus.cc:482] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.499673 6551 raft_consensus.cc:504] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.498461 6668 raft_consensus.cc:504] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.498942 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.507402 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.507782 6470 raft_consensus.cc:2454] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:00.508055 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.509011 6469 raft_consensus.cc:2454] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
W20240503 14:05:00.509264 6436 leader_election.cc:341] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.517805 6364 leader_election.cc:302] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:00.518887 6642 raft_consensus.cc:2781] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.519446 6642 raft_consensus.cc:482] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.519887 6642 raft_consensus.cc:3037] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.524389 6553 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:00.528786 6668 leader_election.cc:288] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.568933 6286 leader_election.cc:302] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:00.576588 6552 raft_consensus.cc:2781] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.577118 6552 raft_consensus.cc:482] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.577595 6552 raft_consensus.cc:3037] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.579625 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:00.581522 6435 leader_election.cc:341] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.582098 6435 leader_election.cc:302] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.584731 6671 raft_consensus.cc:482] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.585403 6671 raft_consensus.cc:504] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.587255 6551 leader_election.cc:288] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.583267 6632 raft_consensus.cc:2726] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:00.599069 6671 leader_election.cc:288] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.598242 6514 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.600314 6514 ts_tablet_manager.cc:1367] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.270s user 0.018s sys 0.022s
I20240503 14:05:00.600725 6544 raft_consensus.cc:504] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.601248 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.603394 6514 raft_consensus.cc:348] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.604221 6514 raft_consensus.cc:374] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.604535 6514 raft_consensus.cc:729] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.609651 6514 consensus_queue.cc:260] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.611418 6544 leader_election.cc:288] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.613198 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:00.613925 6469 raft_consensus.cc:3037] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.616570 6514 ts_tablet_manager.cc:1398] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
I20240503 14:05:00.617209 6632 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:05:00.618036 6436 consensus_peers.cc:485] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:00.619634 6514 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.623327 6322 raft_consensus.cc:2454] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:00.603832 6396 raft_consensus.cc:2454] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:00.628659 6436 leader_election.cc:302] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:00.630199 6632 raft_consensus.cc:2781] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.630738 6632 raft_consensus.cc:686] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:00.631579 6632 consensus_queue.cc:237] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.676215 6551 consensus_queue.cc:1035] T a4080dad66944ea291f6d69736cd2c23 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:05:00.681460 6552 raft_consensus.cc:504] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.688769 6513 tablet_bootstrap.cc:654] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.693614 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.694442 6472 raft_consensus.cc:2454] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:00.693859 6642 raft_consensus.cc:504] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.704615 6397 raft_consensus.cc:2379] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 90a66da3b59e4511bdc1d35930fe558d in this term.
I20240503 14:05:00.704680 6674 raft_consensus.cc:482] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.705474 6674 raft_consensus.cc:504] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.707695 6674 leader_election.cc:288] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.708642 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.710155 6512 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.710646 6512 ts_tablet_manager.cc:1367] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.256s user 0.015s sys 0.011s
I20240503 14:05:00.712426 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:00.703779 6469 raft_consensus.cc:2454] T 0afc2b5c1e62414587bea6a93f52c0c9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:00.713230 6395 raft_consensus.cc:3037] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.713457 6512 raft_consensus.cc:348] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.714269 6512 raft_consensus.cc:374] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.714555 6512 raft_consensus.cc:729] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.715054 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:00.715153 6512 consensus_queue.cc:260] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.715848 6471 raft_consensus.cc:3037] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.712664 6552 leader_election.cc:288] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.717837 6512 ts_tablet_manager.cc:1398] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240503 14:05:00.726636 6512 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.728469 6642 leader_election.cc:288] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.744683 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.745594 6319 raft_consensus.cc:2454] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:00.747201 6514 tablet_bootstrap.cc:654] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.747952 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:00.749390 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef212857f4874110a036926374b4d0cd" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:00.751574 6320 raft_consensus.cc:3037] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.760849 6471 raft_consensus.cc:2454] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:00.738224 6364 leader_election.cc:302] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:00.768869 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801af7dc07784c89890730a686553023" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.770970 6512 tablet_bootstrap.cc:654] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.779536 6219 catalog_manager.cc:5484] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:00.774431 6642 raft_consensus.cc:2781] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.777594 6364 leader_election.cc:302] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:00.784624 6668 raft_consensus.cc:2781] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.785132 6668 raft_consensus.cc:686] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
W20240503 14:05:00.778299 6362 leader_election.cc:341] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.785970 6668 consensus_queue.cc:237] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:00.789018 6436 consensus_peers.cc:485] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:00.790179 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7257be676307482195052c59227129f0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:00.790844 6319 raft_consensus.cc:3037] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.790868 6470 raft_consensus.cc:3037] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.792718 6395 raft_consensus.cc:2454] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:00.796428 6513 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:00.797470 6513 ts_tablet_manager.cc:1367] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.306s user 0.009s sys 0.012s
I20240503 14:05:00.797312 6286 leader_election.cc:302] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:00.798395 6552 raft_consensus.cc:2781] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.798979 6552 raft_consensus.cc:686] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:00.799785 6289 leader_election.cc:302] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:00.801548 6552 consensus_queue.cc:237] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.802575 6679 raft_consensus.cc:482] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:00.795531 6642 raft_consensus.cc:686] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:00.813491 6642 consensus_queue.cc:237] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.816289 6679 raft_consensus.cc:504] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.816542 6220 catalog_manager.cc:5484] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:00.818944 6679 leader_election.cc:288] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.820199 6470 raft_consensus.cc:2454] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:00.820363 6513 raft_consensus.cc:348] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.821147 6513 raft_consensus.cc:374] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.821436 6513 raft_consensus.cc:729] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.822077 6513 consensus_queue.cc:260] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.822665 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2fec4b506c4846935f68b456479e1d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.823918 6320 raft_consensus.cc:2454] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:00.824817 6674 raft_consensus.cc:2741] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:05:00.826071 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2fec4b506c4846935f68b456479e1d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.827455 6544 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:05:00.827962 6436 leader_election.cc:341] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.828045 6544 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.836477 6396 raft_consensus.cc:2454] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:00.837697 6289 leader_election.cc:302] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:00.839080 6513 ts_tablet_manager.cc:1398] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.041s user 0.006s sys 0.002s
I20240503 14:05:00.840039 6513 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:00.796660 6632 consensus_queue.cc:1035] T 0e49c03a6cf8400b98ce8d56e55089e1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20240503 14:05:00.839080 6435 leader_election.cc:302] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.852869 6680 raft_consensus.cc:2781] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:00.853482 6680 raft_consensus.cc:482] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.853937 6680 raft_consensus.cc:3037] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.860882 6544 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.862771 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.863723 6470 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 2.
I20240503 14:05:00.866040 6364 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:00.870764 6668 raft_consensus.cc:2781] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:05:00.871706 6668 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:00.872138 6668 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:05:00.872574 6514 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:00.874114 6514 ts_tablet_manager.cc:1367] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.255s user 0.020s sys 0.016s
I20240503 14:05:00.892297 6514 raft_consensus.cc:348] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.893045 6514 raft_consensus.cc:374] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.893326 6514 raft_consensus.cc:729] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.893954 6514 consensus_queue.cc:260] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.897555 6514 ts_tablet_manager.cc:1398] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20240503 14:05:00.898468 6514 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:00.907368 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:00.910756 6680 raft_consensus.cc:504] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.922106 6680 leader_election.cc:288] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:00.923754 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2fec4b506c4846935f68b456479e1d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:00.924552 6396 raft_consensus.cc:3037] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:00.922941 6552 raft_consensus.cc:482] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8e7d4e1e07e94f21b75ba42bf86334e1)
I20240503 14:05:00.929070 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb2fec4b506c4846935f68b456479e1d" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:00.930871 6436 leader_election.cc:341] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:00.926374 6218 catalog_manager.cc:5484] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:00.934527 6396 raft_consensus.cc:2454] T fb2fec4b506c4846935f68b456479e1d P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:00.937345 6435 leader_election.cc:302] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:00.938777 6552 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.940776 6670 raft_consensus.cc:2726] T 801af7dc07784c89890730a686553023 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:00.946105 6552 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.942361 6680 raft_consensus.cc:2781] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:00.947357 6680 raft_consensus.cc:686] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:00.948448 6680 consensus_queue.cc:237] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.946964 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:00.954136 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:00.954917 6470 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 2.
I20240503 14:05:00.961957 6512 tablet_bootstrap.cc:492] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:00.962303 6514 tablet_bootstrap.cc:654] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.962479 6512 ts_tablet_manager.cc:1367] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.236s user 0.011s sys 0.019s
I20240503 14:05:00.928261 6668 raft_consensus.cc:504] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.965484 6396 raft_consensus.cc:2354] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d for earlier term 2. Current term is 3.
I20240503 14:05:00.967782 6286 leader_election.cc:398] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097) with higher term. Message: Invalid argument: T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d for earlier term 2. Current term is 3.
I20240503 14:05:00.968313 6286 leader_election.cc:401] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240503 14:05:00.969206 6552 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Advancing to term 3
I20240503 14:05:00.970669 6512 raft_consensus.cc:348] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.971616 6512 raft_consensus.cc:374] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:00.971985 6512 raft_consensus.cc:729] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:00.981161 6513 tablet_bootstrap.cc:654] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:00.972801 6512 consensus_queue.cc:260] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:00.985792 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:00.988457 6512 ts_tablet_manager.cc:1398] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.026s user 0.005s sys 0.004s
I20240503 14:05:00.989717 6512 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:00.990819 6668 leader_election.cc:288] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 3 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:00.992655 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c17c5d0f3d64427a915fdfd46f9d91f" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:00.997543 6470 raft_consensus.cc:3037] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:05:00.995509 6552 raft_consensus.cc:2726] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097) with higher term. Message: Invalid argument: T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d for earlier term 2. Current term is 3.
I20240503 14:05:01.011533 6320 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 3.
I20240503 14:05:01.013590 6362 leader_election.cc:302] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:01.015718 6470 raft_consensus.cc:2454] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 3.
I20240503 14:05:01.017436 6568 raft_consensus.cc:2781] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:05:01.021382 6514 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:01.021957 6514 ts_tablet_manager.cc:1367] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.009s
I20240503 14:05:01.022189 6568 raft_consensus.cc:686] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [term 3 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:01.023121 6512 tablet_bootstrap.cc:654] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.023373 6568 consensus_queue.cc:237] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.025810 6514 raft_consensus.cc:348] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.026585 6514 raft_consensus.cc:374] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.027208 6514 raft_consensus.cc:729] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.027904 6514 consensus_queue.cc:260] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.029863 6513 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.032131 6513 ts_tablet_manager.cc:1367] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.007s
I20240503 14:05:01.035992 6513 raft_consensus.cc:348] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.036770 6513 raft_consensus.cc:374] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.037070 6513 raft_consensus.cc:729] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.037881 6513 consensus_queue.cc:260] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.043076 6513 ts_tablet_manager.cc:1398] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240503 14:05:01.044196 6513 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:01.043881 6219 catalog_manager.cc:5484] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:01.030189 6514 ts_tablet_manager.cc:1398] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:05:01.052007 6514 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
W20240503 14:05:01.051138 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:01.055207 6513 tablet_bootstrap.cc:654] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.062987 6680 raft_consensus.cc:482] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.063545 6680 raft_consensus.cc:504] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.069875 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.072500 6396 raft_consensus.cc:2454] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:01.074062 6435 leader_election.cc:302] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:01.075196 6551 raft_consensus.cc:2781] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:01.075675 6551 raft_consensus.cc:482] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:01.076102 6551 raft_consensus.cc:3037] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.071569 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:01.080659 6436 leader_election.cc:341] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.081854 6512 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:01.082475 6512 ts_tablet_manager.cc:1367] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.093s user 0.021s sys 0.006s
I20240503 14:05:01.081871 6680 leader_election.cc:288] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:01.085631 6512 raft_consensus.cc:348] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.086381 6512 raft_consensus.cc:374] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.086673 6512 raft_consensus.cc:729] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.087334 6512 consensus_queue.cc:260] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.089849 6512 ts_tablet_manager.cc:1398] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:05:01.090894 6553 consensus_queue.cc:1035] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.094714 6514 tablet_bootstrap.cc:654] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.097121 6512 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:01.099642 6220 catalog_manager.cc:5484] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:01.106207 6551 raft_consensus.cc:504] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.114256 6551 leader_election.cc:288] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:01.114910 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:01.116672 6436 leader_election.cc:341] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.118630 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:01.119393 6512 tablet_bootstrap.cc:654] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.119488 6395 raft_consensus.cc:3037] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.126742 6513 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.127337 6513 ts_tablet_manager.cc:1367] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.083s user 0.035s sys 0.025s
I20240503 14:05:01.129256 6395 raft_consensus.cc:2454] T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:01.129983 6513 raft_consensus.cc:348] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.130753 6513 raft_consensus.cc:374] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.130990 6513 raft_consensus.cc:729] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.131508 6513 consensus_queue.cc:260] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.130774 6435 leader_election.cc:302] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.136997 6553 raft_consensus.cc:2781] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:01.137638 6553 raft_consensus.cc:686] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:01.138191 6513 ts_tablet_manager.cc:1398] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:05:01.139626 6553 consensus_queue.cc:237] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.141733 6513 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:01.144737 6552 raft_consensus.cc:482] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.145608 6552 raft_consensus.cc:504] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.148792 6552 leader_election.cc:288] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.148661 6679 consensus_queue.cc:1035] T 6985cc0483c34fcd95b3e56b8b59bfb4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.150816 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.152032 6470 raft_consensus.cc:2454] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:01.153628 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.156868 6289 leader_election.cc:302] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:01.157606 6679 raft_consensus.cc:482] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.158233 6679 raft_consensus.cc:504] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.158259 6552 raft_consensus.cc:2781] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:01.160961 6552 raft_consensus.cc:482] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:01.161468 6552 raft_consensus.cc:3037] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.165602 6679 leader_election.cc:288] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
W20240503 14:05:01.169837 6286 leader_election.cc:341] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.171675 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.169035 6513 tablet_bootstrap.cc:654] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:01.173357 6435 leader_election.cc:341] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.170912 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:01.175774 6436 leader_election.cc:341] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.176512 6436 leader_election.cc:302] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.177740 6679 raft_consensus.cc:2726] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.196277 6219 catalog_manager.cc:5484] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 1 to 3, leader changed from 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 3 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:01.196633 6552 raft_consensus.cc:504] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.212796 6514 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:01.213402 6514 ts_tablet_manager.cc:1367] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.162s user 0.007s sys 0.012s
I20240503 14:05:01.214349 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:05:01.216230 6286 leader_election.cc:341] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.216655 6512 tablet_bootstrap.cc:492] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:01.217214 6512 ts_tablet_manager.cc:1367] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.007s
I20240503 14:05:01.220028 6513 tablet_bootstrap.cc:492] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.220620 6513 ts_tablet_manager.cc:1367] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.006s
I20240503 14:05:01.220229 6512 raft_consensus.cc:348] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.221159 6512 raft_consensus.cc:374] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.221446 6512 raft_consensus.cc:729] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.222397 6674 raft_consensus.cc:482] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.222347 6512 consensus_queue.cc:260] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.223044 6674 raft_consensus.cc:504] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.224380 6513 raft_consensus.cc:348] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.225128 6513 raft_consensus.cc:374] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.224712 6512 ts_tablet_manager.cc:1398] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240503 14:05:01.225458 6513 raft_consensus.cc:729] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.226210 6674 leader_election.cc:288] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.226424 6512 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:01.226963 6513 consensus_queue.cc:260] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.224712 6514 raft_consensus.cc:348] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.228621 6514 raft_consensus.cc:374] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.228922 6514 raft_consensus.cc:729] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.229956 6514 consensus_queue.cc:260] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.230693 6513 ts_tablet_manager.cc:1398] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20240503 14:05:01.231689 6513 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:01.231312 6220 catalog_manager.cc:5484] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:01.232892 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:05:01.236492 6289 leader_election.cc:341] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.240592 6568 raft_consensus.cc:482] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.241222 6568 raft_consensus.cc:504] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.244221 6568 leader_election.cc:288] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.244535 6513 tablet_bootstrap.cc:654] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.246287 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.247406 6471 raft_consensus.cc:2454] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:01.247679 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:01.250717 6286 leader_election.cc:341] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.251057 6364 leader_election.cc:302] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:01.251426 6286 leader_election.cc:302] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:01.252526 6668 raft_consensus.cc:2781] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:01.253001 6668 raft_consensus.cc:482] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:01.253157 6514 ts_tablet_manager.cc:1398] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.039s user 0.003s sys 0.005s
I20240503 14:05:01.253445 6668 raft_consensus.cc:3037] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.253895 6670 raft_consensus.cc:2726] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.239214 6589 raft_consensus.cc:482] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.256294 6589 raft_consensus.cc:504] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.259274 6589 leader_election.cc:288] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.262270 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.263770 6642 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.268100 6642 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.254168 6514 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:01.249974 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:01.273766 6320 raft_consensus.cc:2379] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:01.275804 6552 leader_election.cc:288] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.291116 6514 tablet_bootstrap.cc:654] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.296216 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4080dad66944ea291f6d69736cd2c23" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.304996 6286 leader_election.cc:302] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:01.305629 6512 tablet_bootstrap.cc:654] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.305258 6568 consensus_queue.cc:1035] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.306216 6674 raft_consensus.cc:2726] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.309690 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:01.310427 6513 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.310468 6470 raft_consensus.cc:3037] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.311061 6513 ts_tablet_manager.cc:1367] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.080s user 0.022s sys 0.021s
I20240503 14:05:01.326829 6668 raft_consensus.cc:504] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.333248 6513 raft_consensus.cc:348] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.335263 6513 raft_consensus.cc:374] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.335556 6513 raft_consensus.cc:729] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.336213 6513 consensus_queue.cc:260] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.338434 6513 ts_tablet_manager.cc:1398] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20240503 14:05:01.338832 6320 raft_consensus.cc:3037] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.339412 6513 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:01.341271 6668 leader_election.cc:288] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.355667 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:01.356642 6319 raft_consensus.cc:2379] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:01.357327 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d5e8c5c12a43479ecc3382c9576c60" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:01.359536 6679 consensus_queue.cc:1035] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.363639 6364 leader_election.cc:302] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.367875 6514 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:01.368458 6514 ts_tablet_manager.cc:1367] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.011s
I20240503 14:05:01.368391 6544 raft_consensus.cc:2726] T 66d5e8c5c12a43479ecc3382c9576c60 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.372054 6470 raft_consensus.cc:2454] T 66d5e8c5c12a43479ecc3382c9576c60 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:01.374236 6289 leader_election.cc:302] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:01.376224 6552 consensus_queue.cc:1035] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.377419 6674 raft_consensus.cc:2781] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:01.378037 6674 raft_consensus.cc:686] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:01.379151 6674 consensus_queue.cc:237] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:01.401890 6436 consensus_peers.cc:485] T fb2fec4b506c4846935f68b456479e1d P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:01.404408 6514 raft_consensus.cc:348] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.405282 6514 raft_consensus.cc:374] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.405468 6514 raft_consensus.cc:729] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.406312 6512 tablet_bootstrap.cc:492] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:01.406932 6512 ts_tablet_manager.cc:1367] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.181s user 0.022s sys 0.005s
I20240503 14:05:01.406502 6514 consensus_queue.cc:260] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.409276 6514 ts_tablet_manager.cc:1398] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20240503 14:05:01.410491 6514 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:01.412541 6642 consensus_queue.cc:1035] T 0afc2b5c1e62414587bea6a93f52c0c9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:05:01.417100 6552 consensus_queue.cc:1035] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.421705 6513 tablet_bootstrap.cc:654] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.428895 6552 raft_consensus.cc:482] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.429632 6552 raft_consensus.cc:504] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.433228 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.433606 6512 raft_consensus.cc:348] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.434352 6512 raft_consensus.cc:374] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.434639 6512 raft_consensus.cc:729] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.435258 6512 consensus_queue.cc:260] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.450114 6512 ts_tablet_manager.cc:1398] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.043s user 0.002s sys 0.005s
I20240503 14:05:01.451328 6512 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:01.461198 6514 tablet_bootstrap.cc:654] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.460135 6552 leader_election.cc:288] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.542695 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0afc2b5c1e62414587bea6a93f52c0c9" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.542119 6642 raft_consensus.cc:482] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.545614 6642 raft_consensus.cc:504] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.547598 6642 leader_election.cc:288] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.544525 6289 leader_election.cc:302] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:01.552248 6513 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.552744 6513 ts_tablet_manager.cc:1367] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.214s user 0.032s sys 0.070s
I20240503 14:05:01.553179 6544 raft_consensus.cc:482] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.547897 6221 catalog_manager.cc:5484] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:01.553753 6544 raft_consensus.cc:504] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.554345 6674 raft_consensus.cc:2726] T 0afc2b5c1e62414587bea6a93f52c0c9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:05:01.556484 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:01.557385 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:01.559023 6544 leader_election.cc:288] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.559499 6513 raft_consensus.cc:348] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.560488 6513 raft_consensus.cc:374] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.560789 6513 raft_consensus.cc:729] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.561460 6513 consensus_queue.cc:260] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.563779 6513 ts_tablet_manager.cc:1398] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:05:01.564703 6513 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
W20240503 14:05:01.568982 6362 leader_election.cc:341] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.574141 6469 raft_consensus.cc:1261] T 3c17c5d0f3d64427a915fdfd46f9d91f P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 3 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:05:01.574666 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.575572 6470 raft_consensus.cc:2454] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:01.578804 6364 leader_election.cc:302] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.579486 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.580483 6668 raft_consensus.cc:2781] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:01.580382 6568 consensus_queue.cc:1035] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:01.581053 6668 raft_consensus.cc:482] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:01.581534 6668 raft_consensus.cc:3037] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.586723 6322 raft_consensus.cc:3037] T a4080dad66944ea291f6d69736cd2c23 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:01.590713 6362 leader_election.cc:341] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.580441 6469 raft_consensus.cc:2454] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:01.597679 6668 raft_consensus.cc:504] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.600574 6668 leader_election.cc:288] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.603528 6512 tablet_bootstrap.cc:654] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.605139 6364 leader_election.cc:302] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.605549 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:01.607707 6323 raft_consensus.cc:1261] T 3c17c5d0f3d64427a915fdfd46f9d91f P 9a9ba562b78c4746b83a40ca7c30c00d [term 3 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240503 14:05:01.609263 6544 raft_consensus.cc:2781] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:01.610587 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:01.610956 6544 raft_consensus.cc:482] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:01.611398 6471 raft_consensus.cc:3037] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.611416 6544 raft_consensus.cc:3037] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.618494 6551 raft_consensus.cc:482] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.619211 6551 raft_consensus.cc:504] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:01.622848 6362 leader_election.cc:341] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.637125 6668 consensus_queue.cc:1035] T 3c17c5d0f3d64427a915fdfd46f9d91f P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:01.640497 6553 consensus_queue.cc:1035] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:01.643687 6551 leader_election.cc:288] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:01.647774 6513 tablet_bootstrap.cc:654] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.639899 6471 raft_consensus.cc:2454] T fe2d57a792584b2ca83a4a8747a1f1c5 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:01.658532 6642 raft_consensus.cc:482] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.663169 6642 raft_consensus.cc:504] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.664039 6364 leader_election.cc:302] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.664594 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.665328 6642 leader_election.cc:288] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.665966 6642 raft_consensus.cc:2781] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:01.665926 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:01.673384 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:01.673578 6435 leader_election.cc:341] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:01.676362 6362 leader_election.cc:341] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.680182 6514 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:01.681059 6514 ts_tablet_manager.cc:1367] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.271s user 0.026s sys 0.026s
I20240503 14:05:01.685712 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:01.668982 6642 raft_consensus.cc:686] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
W20240503 14:05:01.688724 6436 leader_election.cc:341] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.689410 6436 leader_election.cc:302] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.693019 6553 raft_consensus.cc:2726] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.693679 6642 consensus_queue.cc:237] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.698305 6514 raft_consensus.cc:348] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.699180 6514 raft_consensus.cc:374] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.699493 6514 raft_consensus.cc:729] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.700353 6514 consensus_queue.cc:260] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.703861 6514 ts_tablet_manager.cc:1398] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
W20240503 14:05:01.695227 6364 leader_election.cc:341] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.705942 6514 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:01.718442 6320 raft_consensus.cc:3037] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.711485 6364 leader_election.cc:302] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
W20240503 14:05:01.801275 6362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 47863.9us
I20240503 14:05:01.822118 6544 raft_consensus.cc:504] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.817725 6553 raft_consensus.cc:482] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:01.833870 6553 raft_consensus.cc:504] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.834630 6544 leader_election.cc:288] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:05:01.833781 6722 log.cc:924] Time spent T 4a2f4c7143ea4d49b2be0871738a9a50 P 90a66da3b59e4511bdc1d35930fe558d: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240503 14:05:01.843835 6553 leader_election.cc:288] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
W20240503 14:05:01.844673 6725 log.cc:924] Time spent T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240503 14:05:01.848671 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.855270 6544 raft_consensus.cc:2726] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:01.863750 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:01.876520 6218 catalog_manager.cc:5484] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:01.880954 6512 tablet_bootstrap.cc:492] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:01.881418 6512 ts_tablet_manager.cc:1367] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.430s user 0.017s sys 0.007s
W20240503 14:05:01.869768 6435 leader_election.cc:341] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.883872 6671 raft_consensus.cc:482] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:05:01.891544 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 196346us
I20240503 14:05:01.894726 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:01.895565 6471 raft_consensus.cc:3037] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:01.885363 6671 raft_consensus.cc:504] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.894783 6513 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:01.902387 6513 ts_tablet_manager.cc:1367] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.338s user 0.020s sys 0.022s
I20240503 14:05:01.903609 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:01.905587 6513 raft_consensus.cc:348] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.906387 6513 raft_consensus.cc:374] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.906694 6513 raft_consensus.cc:729] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.907090 6671 leader_election.cc:288] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:01.911473 6471 raft_consensus.cc:2454] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:01.907394 6513 consensus_queue.cc:260] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.911756 6512 raft_consensus.cc:348] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.912930 6512 raft_consensus.cc:374] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:01.913224 6512 raft_consensus.cc:729] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:01.913848 6512 consensus_queue.cc:260] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.916267 6364 leader_election.cc:302] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:01.917696 6728 raft_consensus.cc:2781] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:01.918352 6728 raft_consensus.cc:686] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
W20240503 14:05:01.890206 6436 leader_election.cc:341] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.920606 6436 leader_election.cc:302] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:01.923169 6513 ts_tablet_manager.cc:1398] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.020s user 0.006s sys 0.002s
I20240503 14:05:01.924206 6513 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:01.924842 6514 tablet_bootstrap.cc:654] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.919474 6728 consensus_queue.cc:237] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:01.931771 6513 tablet_bootstrap.cc:654] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:01.934756 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:01.942710 6435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 53328.8us
I20240503 14:05:01.952001 6512 ts_tablet_manager.cc:1398] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.070s user 0.007s sys 0.001s
I20240503 14:05:01.953276 6512 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:01.956275 6727 raft_consensus.cc:2726] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:05:01.957659 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 68899us
I20240503 14:05:01.969921 6551 consensus_queue.cc:1035] T 4a2f4c7143ea4d49b2be0871738a9a50 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240503 14:05:01.977114 6362 leader_election.cc:341] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:01.992636 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a2f4c7143ea4d49b2be0871738a9a50" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.004959 6512 tablet_bootstrap.cc:654] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.005870 6552 consensus_queue.cc:1035] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:02.006042 6289 leader_election.cc:302] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:02.012530 6221 catalog_manager.cc:5484] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:02.020174 6513 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.027007 6514 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.027554 6514 ts_tablet_manager.cc:1367] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.322s user 0.049s sys 0.078s
I20240503 14:05:02.020730 6513 ts_tablet_manager.cc:1367] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.097s user 0.015s sys 0.020s
I20240503 14:05:02.039726 6512 tablet_bootstrap.cc:492] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.041172 6512 ts_tablet_manager.cc:1367] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.088s user 0.018s sys 0.008s
I20240503 14:05:02.046589 6512 raft_consensus.cc:348] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.049685 6552 raft_consensus.cc:2726] T 4a2f4c7143ea4d49b2be0871738a9a50 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:05:02.055125 6512 raft_consensus.cc:374] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.058388 6589 consensus_queue.cc:1035] T 66d5e8c5c12a43479ecc3382c9576c60 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:02.069119 6513 raft_consensus.cc:348] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.071148 6513 raft_consensus.cc:374] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.033579 6514 raft_consensus.cc:348] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.072415 6514 raft_consensus.cc:374] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.075107 6514 raft_consensus.cc:729] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.072866 6513 raft_consensus.cc:729] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.055703 6512 raft_consensus.cc:729] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.076784 6513 consensus_queue.cc:260] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.078121 6514 consensus_queue.cc:260] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.077772 6512 consensus_queue.cc:260] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.086731 6512 ts_tablet_manager.cc:1398] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20240503 14:05:02.087766 6513 ts_tablet_manager.cc:1398] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20240503 14:05:02.102728 6513 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.088521 6514 ts_tablet_manager.cc:1398] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.061s user 0.008s sys 0.000s
I20240503 14:05:02.108422 6514 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:02.117628 6513 tablet_bootstrap.cc:654] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.090574 6512 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.118110 6514 tablet_bootstrap.cc:654] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.136538 6512 tablet_bootstrap.cc:654] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.151202 6724 raft_consensus.cc:482] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240503 14:05:02.151492 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:02.151844 6724 raft_consensus.cc:504] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.154141 6724 leader_election.cc:288] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.155613 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.155711 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:05:02.158212 6364 leader_election.cc:341] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:02.160784 6362 leader_election.cc:341] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.161366 6362 leader_election.cc:302] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.162393 6724 raft_consensus.cc:2726] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:02.166795 6512 tablet_bootstrap.cc:492] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.167369 6512 ts_tablet_manager.cc:1367] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.004s
I20240503 14:05:02.177800 6724 raft_consensus.cc:482] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.178503 6724 raft_consensus.cc:504] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.182258 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.182564 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:05:02.184388 6362 leader_election.cc:341] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:02.185403 6364 leader_election.cc:341] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.188999 6364 leader_election.cc:302] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.189853 6724 leader_election.cc:288] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.191004 6568 raft_consensus.cc:2726] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:02.194849 6513 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.195575 6513 ts_tablet_manager.cc:1367] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.004s
I20240503 14:05:02.207747 6512 raft_consensus.cc:348] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.211258 6512 raft_consensus.cc:374] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.210917 6514 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.210094 6513 raft_consensus.cc:348] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.211988 6514 ts_tablet_manager.cc:1367] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.003s
I20240503 14:05:02.212357 6513 raft_consensus.cc:374] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.212720 6513 raft_consensus.cc:729] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.213490 6513 consensus_queue.cc:260] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.215140 6512 raft_consensus.cc:729] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.216361 6512 consensus_queue.cc:260] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.220310 6513 ts_tablet_manager.cc:1398] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.024s user 0.004s sys 0.004s
I20240503 14:05:02.221355 6513 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.228204 6513 tablet_bootstrap.cc:654] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.229688 6512 ts_tablet_manager.cc:1398] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.062s user 0.007s sys 0.003s
I20240503 14:05:02.231190 6512 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.229676 6724 consensus_queue.cc:1035] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:05:02.240286 6512 tablet_bootstrap.cc:654] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.245863 6514 raft_consensus.cc:348] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.246732 6514 raft_consensus.cc:374] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.247020 6514 raft_consensus.cc:729] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.247813 6514 consensus_queue.cc:260] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.251055 6514 ts_tablet_manager.cc:1398] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20240503 14:05:02.252147 6514 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:02.255367 6513 tablet_bootstrap.cc:492] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.255895 6513 ts_tablet_manager.cc:1367] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.002s
I20240503 14:05:02.257714 6512 tablet_bootstrap.cc:492] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.259140 6512 ts_tablet_manager.cc:1367] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.005s
I20240503 14:05:02.259622 6513 raft_consensus.cc:348] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.261184 6513 raft_consensus.cc:374] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.261395 6513 raft_consensus.cc:729] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.261837 6513 consensus_queue.cc:260] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.266158 6512 raft_consensus.cc:348] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.269858 6512 raft_consensus.cc:374] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.270277 6512 raft_consensus.cc:729] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.269493 6514 tablet_bootstrap.cc:654] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.271167 6320 raft_consensus.cc:3037] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.279136 6512 consensus_queue.cc:260] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.281931 6513 ts_tablet_manager.cc:1398] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.026s user 0.006s sys 0.002s
I20240503 14:05:02.294931 6319 raft_consensus.cc:3037] T 0e49c03a6cf8400b98ce8d56e55089e1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.292625 6512 ts_tablet_manager.cc:1398] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.033s user 0.005s sys 0.004s
I20240503 14:05:02.294399 6552 raft_consensus.cc:482] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.297519 6512 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.304669 6512 tablet_bootstrap.cc:654] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.297005 6552 raft_consensus.cc:504] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.292532 6514 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.307476 6514 ts_tablet_manager.cc:1367] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.003s
I20240503 14:05:02.296468 6513 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.308032 6724 consensus_queue.cc:1035] T fe2d57a792584b2ca83a4a8747a1f1c5 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:02.309870 6544 raft_consensus.cc:482] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.310206 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.311357 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2d57a792584b2ca83a4a8747a1f1c5" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:02.312577 6552 leader_election.cc:288] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.312852 6514 raft_consensus.cc:348] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.313496 6514 raft_consensus.cc:374] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.313709 6514 raft_consensus.cc:729] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.314314 6514 consensus_queue.cc:260] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.316120 6514 ts_tablet_manager.cc:1398] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:05:02.316969 6514 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:02.318771 6286 leader_election.cc:302] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:02.318409 6544 raft_consensus.cc:504] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.320716 6552 raft_consensus.cc:2726] T fe2d57a792584b2ca83a4a8747a1f1c5 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:05:02.322075 6544 leader_election.cc:288] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.323535 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.324426 6471 raft_consensus.cc:2454] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:02.323192 6513 tablet_bootstrap.cc:654] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.325706 6364 leader_election.cc:302] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:02.330338 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.331238 6320 raft_consensus.cc:2454] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:02.332720 6724 raft_consensus.cc:2781] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.335214 6589 raft_consensus.cc:482] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.342595 6514 tablet_bootstrap.cc:654] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.340365 6724 raft_consensus.cc:482] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.350121 6724 raft_consensus.cc:3037] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.344650 6589 raft_consensus.cc:504] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.364411 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:02.365257 6322 raft_consensus.cc:3037] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.365986 6724 raft_consensus.cc:504] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.384860 6323 raft_consensus.cc:3037] T fb2fec4b506c4846935f68b456479e1d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.394660 6724 leader_election.cc:288] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.402045 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:02.402859 6470 raft_consensus.cc:3037] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.404206 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17974ef0de7408c9a2fcebd1f42e07b" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:02.407043 6322 raft_consensus.cc:3037] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.409682 6568 consensus_queue.cc:1035] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:02.412865 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a315c69a72a240f0a5be1b1b8f3625e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.413929 6471 raft_consensus.cc:2379] T a315c69a72a240f0a5be1b1b8f3625e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 90a66da3b59e4511bdc1d35930fe558d in this term.
I20240503 14:05:02.416970 6289 leader_election.cc:302] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:02.423813 6589 leader_election.cc:288] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.434975 6512 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.435505 6512 ts_tablet_manager.cc:1367] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.138s user 0.019s sys 0.021s
I20240503 14:05:02.436879 6470 raft_consensus.cc:2454] T e17974ef0de7408c9a2fcebd1f42e07b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:02.444892 6514 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.445474 6514 ts_tablet_manager.cc:1367] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.129s user 0.015s sys 0.020s
I20240503 14:05:02.445209 6364 leader_election.cc:302] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:02.448657 6568 raft_consensus.cc:2781] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:02.449314 6568 raft_consensus.cc:686] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:02.449110 6514 raft_consensus.cc:348] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.450484 6514 raft_consensus.cc:374] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.453713 6322 raft_consensus.cc:2454] T e17974ef0de7408c9a2fcebd1f42e07b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:02.442677 6512 raft_consensus.cc:348] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.455706 6512 raft_consensus.cc:374] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.450956 6514 raft_consensus.cc:729] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.456060 6512 raft_consensus.cc:729] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.459180 6512 consensus_queue.cc:260] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.450449 6568 consensus_queue.cc:237] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.462517 6514 consensus_queue.cc:260] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.464741 6512 ts_tablet_manager.cc:1398] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.029s user 0.001s sys 0.006s
I20240503 14:05:02.467111 6512 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.473338 6514 ts_tablet_manager.cc:1398] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.028s user 0.006s sys 0.002s
I20240503 14:05:02.474572 6514 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:02.478073 6513 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.478780 6513 ts_tablet_manager.cc:1367] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.183s user 0.009s sys 0.008s
I20240503 14:05:02.480257 6552 raft_consensus.cc:2726] T a315c69a72a240f0a5be1b1b8f3625e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:02.492549 6218 catalog_manager.cc:5484] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:02.495347 6513 raft_consensus.cc:348] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.496419 6513 raft_consensus.cc:374] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.496686 6513 raft_consensus.cc:729] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.502660 6513 consensus_queue.cc:260] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.507098 6513 ts_tablet_manager.cc:1398] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240503 14:05:02.508212 6513 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.514264 6512 tablet_bootstrap.cc:654] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.522913 6724 raft_consensus.cc:482] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.523818 6514 tablet_bootstrap.cc:654] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.525224 6551 raft_consensus.cc:482] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.525854 6551 raft_consensus.cc:504] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.523574 6724 raft_consensus.cc:504] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.528817 6551 leader_election.cc:288] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:02.531500 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.534204 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0562bcbd0a407a9c73885ac89858f9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.534576 6513 tablet_bootstrap.cc:654] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:02.535903 6436 leader_election.cc:341] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.536368 6727 raft_consensus.cc:482] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.537951 6727 raft_consensus.cc:504] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:02.540264 6362 leader_election.cc:341] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.540709 6727 leader_election.cc:288] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:02.537046 6728 consensus_queue.cc:1035] T a315c69a72a240f0a5be1b1b8f3625e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240503 14:05:02.542083 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0562bcbd0a407a9c73885ac89858f9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.542924 6471 raft_consensus.cc:2454] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:02.543702 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:02.544486 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:02.545388 6364 leader_election.cc:302] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.535804 6724 leader_election.cc:288] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.544571 6397 raft_consensus.cc:2454] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:02.546648 6724 raft_consensus.cc:2781] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:05:02.547565 6435 leader_election.cc:341] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.541965 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:02.556954 6436 leader_election.cc:341] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.553689 6724 raft_consensus.cc:482] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.559871 6724 raft_consensus.cc:3037] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.556686 6435 leader_election.cc:302] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.566071 6727 raft_consensus.cc:2726] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:02.567739 6435 leader_election.cc:302] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.585692 6512 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.586233 6512 ts_tablet_manager.cc:1367] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.014s
I20240503 14:05:02.587219 6514 tablet_bootstrap.cc:492] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.587827 6514 ts_tablet_manager.cc:1367] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.006s
I20240503 14:05:02.590636 6512 raft_consensus.cc:348] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.590834 6514 raft_consensus.cc:348] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.591454 6512 raft_consensus.cc:374] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.591688 6514 raft_consensus.cc:374] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.592049 6514 raft_consensus.cc:729] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.593667 6514 consensus_queue.cc:260] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.596194 6724 raft_consensus.cc:504] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.600139 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0562bcbd0a407a9c73885ac89858f9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:02.600627 6514 ts_tablet_manager.cc:1398] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240503 14:05:02.601831 6514 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
W20240503 14:05:02.602106 6362 leader_election.cc:341] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.591843 6512 raft_consensus.cc:729] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.603530 6512 consensus_queue.cc:260] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.608343 6727 raft_consensus.cc:2781] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.608796 6512 ts_tablet_manager.cc:1398] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20240503 14:05:02.610111 6512 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.608886 6727 raft_consensus.cc:482] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.611053 6727 raft_consensus.cc:3037] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:02.616391 6435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 48765us
I20240503 14:05:02.620350 6724 leader_election.cc:288] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.621960 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0562bcbd0a407a9c73885ac89858f9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:02.622713 6470 raft_consensus.cc:3037] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.633149 6512 tablet_bootstrap.cc:654] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.639050 6470 raft_consensus.cc:2454] T 7f0562bcbd0a407a9c73885ac89858f9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:02.640969 6364 leader_election.cc:302] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.641996 6724 raft_consensus.cc:2781] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:02.642509 6724 raft_consensus.cc:686] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:02.643601 6727 raft_consensus.cc:504] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.647234 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:02.648757 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b30b12892024636b275c9070901327b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:02.649438 6395 raft_consensus.cc:3037] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.643760 6724 consensus_queue.cc:237] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:02.650285 6436 leader_election.cc:341] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.658191 6395 raft_consensus.cc:2454] T 2b30b12892024636b275c9070901327b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:02.659734 6435 leader_election.cc:302] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.660916 6679 raft_consensus.cc:2781] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:02.661511 6679 raft_consensus.cc:686] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:02.662532 6679 consensus_queue.cc:237] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.664742 6727 leader_election.cc:288] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:02.667755 6512 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.668321 6512 ts_tablet_manager.cc:1367] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.001s
I20240503 14:05:02.680289 6513 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.678671 6512 raft_consensus.cc:348] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.681919 6513 ts_tablet_manager.cc:1367] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.174s user 0.035s sys 0.026s
I20240503 14:05:02.747324 6512 raft_consensus.cc:374] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.747697 6512 raft_consensus.cc:729] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.737285 6514 tablet_bootstrap.cc:654] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.726886 6221 catalog_manager.cc:5484] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:02.749477 6512 consensus_queue.cc:260] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.732518 6220 catalog_manager.cc:5484] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:02.756049 6513 raft_consensus.cc:348] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.756873 6513 raft_consensus.cc:374] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.757076 6513 raft_consensus.cc:729] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.757537 6513 consensus_queue.cc:260] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.762951 6512 ts_tablet_manager.cc:1398] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.094s user 0.032s sys 0.021s
I20240503 14:05:02.764353 6513 ts_tablet_manager.cc:1398] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20240503 14:05:02.767360 6512 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.782006 6513 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.795892 6514 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.796466 6514 ts_tablet_manager.cc:1367] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.195s user 0.012s sys 0.012s
I20240503 14:05:02.809880 6514 raft_consensus.cc:348] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.810668 6514 raft_consensus.cc:374] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.810966 6514 raft_consensus.cc:729] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.811646 6514 consensus_queue.cc:260] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.815150 6512 tablet_bootstrap.cc:654] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.816845 6724 raft_consensus.cc:482] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.817332 6514 ts_tablet_manager.cc:1398] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240503 14:05:02.817394 6724 raft_consensus.cc:504] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.818624 6514 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:02.822302 6724 leader_election.cc:288] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.824481 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.825259 6470 raft_consensus.cc:2454] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:02.828187 6513 tablet_bootstrap.cc:654] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.830016 6364 leader_election.cc:302] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:02.831002 6568 raft_consensus.cc:2781] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.831396 6568 raft_consensus.cc:482] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.831775 6568 raft_consensus.cc:3037] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.837299 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:02.839406 6362 leader_election.cc:341] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.848932 6759 raft_consensus.cc:482] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.849633 6759 raft_consensus.cc:504] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.853459 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:02.853708 6759 leader_election.cc:288] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:02.854372 6323 raft_consensus.cc:2454] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:02.855808 6436 leader_election.cc:302] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:02.855263 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:02.856782 6551 raft_consensus.cc:2781] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.857179 6551 raft_consensus.cc:482] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.857226 6395 raft_consensus.cc:2454] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:02.857578 6551 raft_consensus.cc:3037] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.863014 6568 raft_consensus.cc:504] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.866967 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:02.869019 6362 leader_election.cc:341] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.874523 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:02.875391 6470 raft_consensus.cc:3037] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.877426 6551 raft_consensus.cc:504] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.882658 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:02.883474 6322 raft_consensus.cc:3037] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.885658 6514 tablet_bootstrap.cc:654] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.892809 6513 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:02.893347 6513 ts_tablet_manager.cc:1367] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.112s user 0.015s sys 0.010s
I20240503 14:05:02.894526 6512 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:02.895118 6512 ts_tablet_manager.cc:1367] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.128s user 0.014s sys 0.022s
I20240503 14:05:02.898712 6512 raft_consensus.cc:348] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.900374 6512 raft_consensus.cc:374] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.902112 6512 raft_consensus.cc:729] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.902868 6512 consensus_queue.cc:260] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.905427 6512 ts_tablet_manager.cc:1398] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 14:05:02.906857 6470 raft_consensus.cc:2454] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:02.907208 6512 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:02.908310 6364 leader_election.cc:302] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.880649 6568 leader_election.cc:288] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.909957 6568 raft_consensus.cc:2781] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:02.910383 6322 raft_consensus.cc:2454] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:02.910723 6551 leader_election.cc:288] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:02.911877 6436 leader_election.cc:302] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:02.912825 6551 raft_consensus.cc:2781] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:02.914783 6551 raft_consensus.cc:686] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:02.913359 6568 raft_consensus.cc:686] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:02.922832 6512 tablet_bootstrap.cc:654] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.924552 6728 consensus_queue.cc:1035] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:02.923545 6551 consensus_queue.cc:237] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.929746 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df77168e3be74202b705c73539c514d0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:02.930531 6395 raft_consensus.cc:3037] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.932580 6513 raft_consensus.cc:348] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.933470 6513 raft_consensus.cc:374] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:02.933786 6513 raft_consensus.cc:729] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:02.934434 6513 consensus_queue.cc:260] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.938012 6513 ts_tablet_manager.cc:1398] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20240503 14:05:02.938967 6513 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:02.922688 6642 raft_consensus.cc:482] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.940439 6642 raft_consensus.cc:504] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.943109 6642 leader_election.cc:288] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.922741 6568 consensus_queue.cc:237] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.944567 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:02.950731 6362 leader_election.cc:341] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.951478 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.952646 6471 raft_consensus.cc:2454] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:02.957973 6364 leader_election.cc:302] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:02.959852 6642 raft_consensus.cc:2781] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.960356 6642 raft_consensus.cc:482] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.963513 6642 raft_consensus.cc:3037] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.969858 6671 raft_consensus.cc:482] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:02.970444 6671 raft_consensus.cc:504] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.972880 6671 leader_election.cc:288] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:02.979915 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:02.980221 6642 raft_consensus.cc:504] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:02.980898 6471 raft_consensus.cc:2454] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:02.981382 6513 tablet_bootstrap.cc:654] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:02.985625 6514 tablet_bootstrap.cc:492] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:02.986248 6514 ts_tablet_manager.cc:1367] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.168s user 0.013s sys 0.010s
I20240503 14:05:02.989341 6289 leader_election.cc:302] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:02.990777 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:02.992363 6286 leader_election.cc:341] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:02.992491 6395 raft_consensus.cc:2454] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:02.994822 6671 raft_consensus.cc:2781] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:02.996414 6671 raft_consensus.cc:482] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:02.996866 6671 raft_consensus.cc:3037] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:02.999053 6220 catalog_manager.cc:5484] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:03.003262 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:03.004823 6362 leader_election.cc:341] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.006063 6642 leader_election.cc:288] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.008216 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f401553cb154845b5d0a890810a9837" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:03.009079 6471 raft_consensus.cc:3037] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.028479 6512 tablet_bootstrap.cc:492] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.029121 6512 ts_tablet_manager.cc:1367] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.122s user 0.020s sys 0.005s
I20240503 14:05:03.036808 6512 raft_consensus.cc:348] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.037724 6512 raft_consensus.cc:374] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.038097 6512 raft_consensus.cc:729] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.038851 6512 consensus_queue.cc:260] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.045050 6642 consensus_queue.cc:1035] T e17974ef0de7408c9a2fcebd1f42e07b P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.046360 6514 raft_consensus.cc:348] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.047794 6514 raft_consensus.cc:374] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.048205 6514 raft_consensus.cc:729] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.015282 6671 raft_consensus.cc:504] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.051797 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:03.043467 6220 catalog_manager.cc:5484] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:03.051913 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157836a198e44425a9f455d4c07fa5c4" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
W20240503 14:05:03.053870 6286 leader_election.cc:341] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.054631 6470 raft_consensus.cc:3037] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.045533 6512 ts_tablet_manager.cc:1398] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20240503 14:05:03.061092 6512 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.068645 6514 consensus_queue.cc:260] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.071154 6514 ts_tablet_manager.cc:1398] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.085s user 0.003s sys 0.004s
I20240503 14:05:03.072353 6514 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.073685 6671 leader_election.cc:288] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.023661 6471 raft_consensus.cc:2454] T 5f401553cb154845b5d0a890810a9837 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:03.084190 6642 consensus_queue.cc:1035] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 14:05:03.095299 6513 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.095826 6513 ts_tablet_manager.cc:1367] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.157s user 0.024s sys 0.022s
I20240503 14:05:03.083952 6364 leader_election.cc:302] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:03.099177 6724 raft_consensus.cc:2781] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:03.099691 6724 raft_consensus.cc:686] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:03.106137 6513 raft_consensus.cc:348] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.107540 6513 raft_consensus.cc:374] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.107833 6724 consensus_queue.cc:237] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.107836 6513 raft_consensus.cc:729] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.103263 6727 consensus_queue.cc:1035] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240503 14:05:03.109690 6513 consensus_queue.cc:260] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.113674 6513 ts_tablet_manager.cc:1398] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20240503 14:05:03.114617 6513 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.122308 6470 raft_consensus.cc:2454] T 157836a198e44425a9f455d4c07fa5c4 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:03.143613 6321 raft_consensus.cc:3037] T 2b30b12892024636b275c9070901327b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.147418 6513 tablet_bootstrap.cc:654] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.172696 6289 leader_election.cc:302] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:03.177251 6671 raft_consensus.cc:2781] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:03.177965 6671 raft_consensus.cc:686] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:03.179085 6671 consensus_queue.cc:237] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:03.206856 6362 consensus_peers.cc:485] T 7f0562bcbd0a407a9c73885ac89858f9 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:03.221787 6552 raft_consensus.cc:482] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.222435 6552 raft_consensus.cc:504] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.223071 6772 raft_consensus.cc:482] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.223709 6772 raft_consensus.cc:504] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.225147 6514 tablet_bootstrap.cc:654] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.225977 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.227185 6395 raft_consensus.cc:2454] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:03.226380 6772 leader_election.cc:288] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.228564 6512 tablet_bootstrap.cc:654] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.231365 6286 leader_election.cc:302] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:03.232144 6552 leader_election.cc:288] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.232599 6772 raft_consensus.cc:2781] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:03.231608 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.234153 6772 raft_consensus.cc:482] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:03.234690 6772 raft_consensus.cc:3037] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:03.235693 6286 leader_election.cc:341] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.236841 6727 consensus_queue.cc:1035] T 2b30b12892024636b275c9070901327b P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:05:03.240221 6513 tablet_bootstrap.cc:492] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.249166 6513 ts_tablet_manager.cc:1367] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.135s user 0.016s sys 0.008s
I20240503 14:05:03.251703 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.252619 6470 raft_consensus.cc:2454] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:03.253940 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.254616 6220 catalog_manager.cc:5484] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:03.258512 6513 raft_consensus.cc:348] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.262054 6513 raft_consensus.cc:374] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.262624 6513 raft_consensus.cc:729] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.263417 6513 consensus_queue.cc:260] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.274610 6513 ts_tablet_manager.cc:1398] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20240503 14:05:03.275820 6513 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.276788 6772 raft_consensus.cc:504] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.283562 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:03.284348 6396 raft_consensus.cc:3037] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.284693 6218 catalog_manager.cc:5484] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:03.304972 6772 leader_election.cc:288] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.321786 6512 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.322650 6512 ts_tablet_manager.cc:1367] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.262s user 0.020s sys 0.006s
I20240503 14:05:03.326910 6513 tablet_bootstrap.cc:654] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:03.332702 6289 leader_election.cc:341] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.333319 6289 leader_election.cc:302] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:03.331972 6396 raft_consensus.cc:2454] T a707858e929f40f89e2be68639a03040 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:03.340523 6286 leader_election.cc:302] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
W20240503 14:05:03.341565 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:03.343650 6514 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:03.344170 6514 ts_tablet_manager.cc:1367] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.272s user 0.024s sys 0.011s
I20240503 14:05:03.344830 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a707858e929f40f89e2be68639a03040" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:03.345566 6469 raft_consensus.cc:3037] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.345736 6513 tablet_bootstrap.cc:492] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.346340 6513 ts_tablet_manager.cc:1367] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.071s user 0.011s sys 0.009s
I20240503 14:05:03.345727 6772 raft_consensus.cc:2726] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:03.345600 6512 raft_consensus.cc:348] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.347934 6512 raft_consensus.cc:374] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.348134 6514 raft_consensus.cc:348] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.348945 6514 raft_consensus.cc:374] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.348266 6512 raft_consensus.cc:729] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.349256 6514 raft_consensus.cc:729] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.345870 6778 raft_consensus.cc:2781] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:03.349942 6514 consensus_queue.cc:260] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.350389 6778 raft_consensus.cc:686] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:03.352310 6514 ts_tablet_manager.cc:1398] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20240503 14:05:03.349377 6513 raft_consensus.cc:348] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.349939 6512 consensus_queue.cc:260] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.353235 6513 raft_consensus.cc:374] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.353554 6513 raft_consensus.cc:729] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.354171 6513 consensus_queue.cc:260] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.355368 6512 ts_tablet_manager.cc:1398] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20240503 14:05:03.356312 6512 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.356552 6513 ts_tablet_manager.cc:1398] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 14:05:03.351518 6778 consensus_queue.cc:237] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.358258 6469 raft_consensus.cc:2454] T a707858e929f40f89e2be68639a03040 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:03.360811 6513 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.362144 6514 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.368830 6514 tablet_bootstrap.cc:654] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.370842 6512 tablet_bootstrap.cc:654] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.376293 6513 tablet_bootstrap.cc:654] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.383304 6512 tablet_bootstrap.cc:492] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.383821 6512 ts_tablet_manager.cc:1367] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s
W20240503 14:05:03.386484 6362 consensus_peers.cc:485] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:03.388864 6512 raft_consensus.cc:348] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.389637 6512 raft_consensus.cc:374] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.389907 6512 raft_consensus.cc:729] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.391083 6512 consensus_queue.cc:260] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.394009 6512 ts_tablet_manager.cc:1398] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:05:03.395027 6512 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.396214 6514 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:03.396890 6514 ts_tablet_manager.cc:1367] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.003s
I20240503 14:05:03.400956 6514 raft_consensus.cc:348] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.401784 6512 tablet_bootstrap.cc:654] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.401911 6514 raft_consensus.cc:374] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.402415 6514 raft_consensus.cc:729] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.403210 6514 consensus_queue.cc:260] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.406127 6514 ts_tablet_manager.cc:1398] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240503 14:05:03.408695 6514 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.414505 6642 raft_consensus.cc:482] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.415091 6642 raft_consensus.cc:504] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.417219 6642 leader_election.cc:288] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.419219 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.419700 6514 tablet_bootstrap.cc:654] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.420053 6469 raft_consensus.cc:2454] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:03.422048 6364 leader_election.cc:302] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:03.425364 6728 raft_consensus.cc:2781] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:03.425853 6728 raft_consensus.cc:482] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:03.426275 6728 raft_consensus.cc:3037] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.429267 6772 raft_consensus.cc:482] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.429912 6772 raft_consensus.cc:504] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.432523 6772 leader_election.cc:288] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.433990 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:03.435782 6286 leader_election.cc:341] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.433924 6221 catalog_manager.cc:5484] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:03.436499 6728 raft_consensus.cc:504] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.441411 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.442194 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:03.442939 6471 raft_consensus.cc:3037] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.443681 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:03.444496 6322 raft_consensus.cc:2454] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:03.454402 6544 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.461498 6727 consensus_queue.cc:1035] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.463825 6512 tablet_bootstrap.cc:492] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.464419 6512 ts_tablet_manager.cc:1367] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.070s user 0.021s sys 0.001s
I20240503 14:05:03.473671 6512 raft_consensus.cc:348] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.474463 6512 raft_consensus.cc:374] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.474766 6512 raft_consensus.cc:729] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.475430 6512 consensus_queue.cc:260] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.479384 6512 ts_tablet_manager.cc:1398] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20240503 14:05:03.480398 6512 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.500808 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4148780059584da6a02aaedbf2dd9cf0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:03.501634 6321 raft_consensus.cc:3037] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.497399 6728 leader_election.cc:288] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:05:03.501049 6289 leader_election.cc:341] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.513314 6471 raft_consensus.cc:2454] T 4148780059584da6a02aaedbf2dd9cf0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:03.510088 6289 leader_election.cc:302] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:03.529681 6364 leader_election.cc:302] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:03.530901 6642 raft_consensus.cc:2781] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:03.531510 6642 raft_consensus.cc:686] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:03.532567 6642 consensus_queue.cc:237] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.534358 6321 raft_consensus.cc:2454] T 4148780059584da6a02aaedbf2dd9cf0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:03.534832 6778 raft_consensus.cc:2726] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:03.551785 6514 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:03.552349 6514 ts_tablet_manager.cc:1367] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.008s
I20240503 14:05:03.553763 6512 tablet_bootstrap.cc:654] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.560829 6513 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.561379 6513 ts_tablet_manager.cc:1367] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.201s user 0.024s sys 0.043s
I20240503 14:05:03.562012 6514 raft_consensus.cc:348] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.562599 6514 raft_consensus.cc:374] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.562800 6514 raft_consensus.cc:729] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.563324 6514 consensus_queue.cc:260] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.552965 6759 consensus_queue.cc:1035] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240503 14:05:03.570502 6514 ts_tablet_manager.cc:1398] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:05:03.572193 6514 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.576066 6513 raft_consensus.cc:348] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.576954 6513 raft_consensus.cc:374] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.577242 6513 raft_consensus.cc:729] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.578006 6513 consensus_queue.cc:260] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.580353 6513 ts_tablet_manager.cc:1398] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20240503 14:05:03.581236 6513 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.596336 6728 raft_consensus.cc:482] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.597767 6728 raft_consensus.cc:504] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.602154 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:03.608126 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.608760 6513 tablet_bootstrap.cc:654] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:03.610204 6364 leader_election.cc:341] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.611196 6514 tablet_bootstrap.cc:654] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.611826 6728 leader_election.cc:288] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.613871 6220 catalog_manager.cc:5484] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
W20240503 14:05:03.611310 6362 leader_election.cc:341] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.616530 6758 raft_consensus.cc:482] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.616662 6362 leader_election.cc:302] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:03.617235 6758 raft_consensus.cc:504] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.624130 6758 leader_election.cc:288] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:03.624514 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:03.625170 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:03.626389 6436 leader_election.cc:341] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:03.627008 6435 leader_election.cc:341] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.627601 6435 leader_election.cc:302] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:03.630502 6323 raft_consensus.cc:3037] T 7f0562bcbd0a407a9c73885ac89858f9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.620061 6728 raft_consensus.cc:2726] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:03.632259 6758 raft_consensus.cc:2726] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:03.639850 6512 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.640847 6512 ts_tablet_manager.cc:1367] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.161s user 0.011s sys 0.012s
I20240503 14:05:03.645294 6512 raft_consensus.cc:348] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.646040 6512 raft_consensus.cc:374] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.646387 6512 raft_consensus.cc:729] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.647346 6512 consensus_queue.cc:260] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.650894 6396 raft_consensus.cc:3037] T 157836a198e44425a9f455d4c07fa5c4 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.672310 6728 consensus_queue.cc:1035] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.673959 6512 ts_tablet_manager.cc:1398] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.033s user 0.003s sys 0.004s
I20240503 14:05:03.674880 6512 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.676295 6671 consensus_queue.cc:1035] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.681315 6589 consensus_queue.cc:1035] T 157836a198e44425a9f455d4c07fa5c4 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.715070 6514 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:03.715554 6514 ts_tablet_manager.cc:1367] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.144s user 0.046s sys 0.011s
I20240503 14:05:03.718618 6514 raft_consensus.cc:348] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.719502 6514 raft_consensus.cc:374] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.719847 6514 raft_consensus.cc:729] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
W20240503 14:05:03.722450 6362 consensus_peers.cc:485] T 5f401553cb154845b5d0a890810a9837 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:03.725414 6514 consensus_queue.cc:260] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.729718 6514 ts_tablet_manager.cc:1398] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240503 14:05:03.730226 6513 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.730737 6513 ts_tablet_manager.cc:1367] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.009s
I20240503 14:05:03.731371 6514 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.738692 6513 raft_consensus.cc:348] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.740737 6513 raft_consensus.cc:374] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.741079 6513 raft_consensus.cc:729] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.741787 6513 consensus_queue.cc:260] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.764022 6513 ts_tablet_manager.cc:1398] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.033s user 0.006s sys 0.002s
I20240503 14:05:03.765069 6513 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.773365 6512 tablet_bootstrap.cc:654] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.784235 6758 raft_consensus.cc:482] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.785164 6758 raft_consensus.cc:504] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.789986 6514 tablet_bootstrap.cc:654] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.791468 6758 leader_election.cc:288] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:03.796975 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.797907 6396 raft_consensus.cc:2454] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:03.799000 6513 tablet_bootstrap.cc:654] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.799579 6589 raft_consensus.cc:482] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.799975 6435 leader_election.cc:302] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:03.800194 6589 raft_consensus.cc:504] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.801934 6679 raft_consensus.cc:2781] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:03.802412 6679 raft_consensus.cc:482] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:03.802847 6589 leader_election.cc:288] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:03.803560 6679 raft_consensus.cc:3037] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.807411 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.807427 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:03.810698 6322 raft_consensus.cc:2454] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:03.816586 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0285fa210834b8c9b116d0cf7bb2430" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:03.820786 6289 leader_election.cc:302] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:03.822346 6772 raft_consensus.cc:2726] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:03.839260 6322 raft_consensus.cc:3037] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.838891 6679 raft_consensus.cc:504] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.842151 6589 consensus_queue.cc:1035] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.849925 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:03.850805 6323 raft_consensus.cc:3037] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.853884 6513 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:03.854494 6513 ts_tablet_manager.cc:1367] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.007s
I20240503 14:05:03.857820 6513 raft_consensus.cc:348] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.858646 6513 raft_consensus.cc:374] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.858954 6513 raft_consensus.cc:729] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.865782 6323 raft_consensus.cc:2454] T 1fd4c415b34640bd82515afe5d3a7632 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:03.868574 6436 leader_election.cc:302] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:03.869577 6758 raft_consensus.cc:2781] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:03.870218 6758 raft_consensus.cc:686] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:03.874293 6512 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.876185 6512 ts_tablet_manager.cc:1367] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.201s user 0.020s sys 0.000s
I20240503 14:05:03.871402 6758 consensus_queue.cc:237] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.887027 6589 consensus_queue.cc:1035] T a707858e929f40f89e2be68639a03040 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.888511 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fd4c415b34640bd82515afe5d3a7632" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:03.889605 6397 raft_consensus.cc:3037] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:03.897270 6512 raft_consensus.cc:348] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.898185 6512 raft_consensus.cc:374] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.898504 6512 raft_consensus.cc:729] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.899173 6512 consensus_queue.cc:260] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.859652 6513 consensus_queue.cc:260] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.911298 6513 ts_tablet_manager.cc:1398] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20240503 14:05:03.912693 6513 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:03.886801 6679 leader_election.cc:288] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:03.917236 6512 ts_tablet_manager.cc:1398] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20240503 14:05:03.918320 6512 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:03.907809 6218 catalog_manager.cc:5484] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:03.924033 6397 raft_consensus.cc:2454] T 1fd4c415b34640bd82515afe5d3a7632 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:03.931473 6512 tablet_bootstrap.cc:654] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.933444 6514 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:03.933972 6514 ts_tablet_manager.cc:1367] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.203s user 0.071s sys 0.045s
I20240503 14:05:03.936424 6514 raft_consensus.cc:348] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.937168 6514 raft_consensus.cc:374] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.937462 6514 raft_consensus.cc:729] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.938102 6514 consensus_queue.cc:260] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.941417 6513 tablet_bootstrap.cc:654] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.941411 6514 ts_tablet_manager.cc:1398] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:05:03.946662 6514 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:03.949594 6679 raft_consensus.cc:482] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.950353 6679 raft_consensus.cc:504] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.954793 6679 leader_election.cc:288] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:03.956910 6512 tablet_bootstrap.cc:492] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:03.959375 6512 ts_tablet_manager.cc:1367] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.000s
I20240503 14:05:03.962693 6512 raft_consensus.cc:348] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.964064 6512 raft_consensus.cc:374] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:03.964452 6512 raft_consensus.cc:729] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:03.965145 6512 consensus_queue.cc:260] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.963584 6679 raft_consensus.cc:482] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:03.966207 6679 raft_consensus.cc:504] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:03.968979 6724 consensus_queue.cc:1035] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:03.969444 6679 leader_election.cc:288] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:03.971683 6512 ts_tablet_manager.cc:1398] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20240503 14:05:03.972290 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.977258 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b840e10839487fac480c2bbf79f329" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:03.979295 6514 tablet_bootstrap.cc:654] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:03.977330 6512 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
W20240503 14:05:03.985816 6435 leader_election.cc:341] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:03.957430 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b840e10839487fac480c2bbf79f329" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:03.994195 6397 raft_consensus.cc:2454] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
W20240503 14:05:03.998368 6436 leader_election.cc:341] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.001442 6435 leader_election.cc:302] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.002784 6758 raft_consensus.cc:2781] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.003239 6758 raft_consensus.cc:482] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.003646 6758 raft_consensus.cc:3037] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.009253 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.013242 6319 raft_consensus.cc:2454] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:04.023231 6512 tablet_bootstrap.cc:654] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.029731 6514 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:04.030175 6436 leader_election.cc:302] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:04.030223 6514 ts_tablet_manager.cc:1367] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.007s
I20240503 14:05:04.032245 6727 raft_consensus.cc:2781] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.032678 6727 raft_consensus.cc:482] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.033033 6727 raft_consensus.cc:3037] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.034638 6514 raft_consensus.cc:348] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.035499 6514 raft_consensus.cc:374] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.035797 6514 raft_consensus.cc:729] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.037881 6513 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:04.037591 6514 consensus_queue.cc:260] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.038372 6513 ts_tablet_manager.cc:1367] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.026s
I20240503 14:05:04.040280 6514 ts_tablet_manager.cc:1398] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:05:04.042665 6513 raft_consensus.cc:348] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.043435 6513 raft_consensus.cc:374] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.043722 6513 raft_consensus.cc:729] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.044478 6513 consensus_queue.cc:260] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.046979 6514 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:04.052922 6513 ts_tablet_manager.cc:1398] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240503 14:05:04.053951 6513 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:04.055104 6728 consensus_queue.cc:1035] T 4148780059584da6a02aaedbf2dd9cf0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.062019 6728 raft_consensus.cc:482] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.062994 6728 raft_consensus.cc:504] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.065837 6728 leader_election.cc:288] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.066044 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:04.067799 6362 leader_election.cc:341] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.069530 6727 raft_consensus.cc:504] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.073068 6551 raft_consensus.cc:482] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.073619 6551 raft_consensus.cc:504] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.078992 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.079332 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:04.080029 6318 raft_consensus.cc:3037] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.080781 6512 tablet_bootstrap.cc:492] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.081271 6512 ts_tablet_manager.cc:1367] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.104s user 0.019s sys 0.010s
I20240503 14:05:04.078585 6727 leader_election.cc:288] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
W20240503 14:05:04.084167 6436 leader_election.cc:341] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.083209 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:04.059803 6758 raft_consensus.cc:504] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:04.087109 6435 leader_election.cc:341] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.087658 6551 leader_election.cc:288] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:04.087818 6758 leader_election.cc:288] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:04.090082 6514 tablet_bootstrap.cc:654] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.092149 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b840e10839487fac480c2bbf79f329" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:04.092489 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:04.093713 6396 raft_consensus.cc:2454] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:04.094688 6513 tablet_bootstrap.cc:654] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.085274 6512 raft_consensus.cc:348] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.096230 6512 raft_consensus.cc:374] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.096516 6512 raft_consensus.cc:729] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.097295 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b840e10839487fac480c2bbf79f329" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:04.101369 6396 raft_consensus.cc:3037] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:04.099172 6436 leader_election.cc:341] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.098301 6435 leader_election.cc:302] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.105011 6318 raft_consensus.cc:2454] T e09711da83964cb1b409403e16852e17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:04.097893 6512 consensus_queue.cc:260] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.106456 6436 leader_election.cc:302] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:04.107174 6512 ts_tablet_manager.cc:1398] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20240503 14:05:04.107573 6551 raft_consensus.cc:2781] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.107959 6551 raft_consensus.cc:482] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.108381 6551 raft_consensus.cc:3037] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.109107 6758 raft_consensus.cc:2781] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.109925 6758 raft_consensus.cc:686] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:04.111016 6758 consensus_queue.cc:237] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.116663 6512 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:04.101840 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.120301 6551 raft_consensus.cc:504] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.128939 6551 leader_election.cc:288] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:04.136270 6513 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:04.150818 6513 ts_tablet_manager.cc:1367] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.014s
I20240503 14:05:04.143435 6512 tablet_bootstrap.cc:654] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.129657 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:04.143986 6471 raft_consensus.cc:2379] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:04.149449 6671 raft_consensus.cc:482] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.161624 6396 raft_consensus.cc:2454] T c9b840e10839487fac480c2bbf79f329 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:04.154189 6671 raft_consensus.cc:504] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.163283 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:04.167212 6397 raft_consensus.cc:3037] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.168677 6318 raft_consensus.cc:3037] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.170593 6514 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:04.171106 6514 ts_tablet_manager.cc:1367] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.124s user 0.018s sys 0.014s
I20240503 14:05:04.172842 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:04.162519 6642 raft_consensus.cc:482] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.173746 6642 raft_consensus.cc:504] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.175880 6671 leader_election.cc:288] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.173944 6514 raft_consensus.cc:348] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.177870 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.178339 6514 raft_consensus.cc:374] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.178642 6514 raft_consensus.cc:729] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
W20240503 14:05:04.179576 6362 leader_election.cc:341] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.179913 6364 leader_election.cc:302] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.182214 6514 consensus_queue.cc:260] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.181239 6221 catalog_manager.cc:5484] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:04.184273 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.184700 6318 raft_consensus.cc:2454] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:04.185647 6471 raft_consensus.cc:2454] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:04.186913 6514 ts_tablet_manager.cc:1398] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.015s user 0.002s sys 0.005s
I20240503 14:05:04.165983 6758 raft_consensus.cc:482] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.198324 6758 raft_consensus.cc:504] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.199373 6513 raft_consensus.cc:348] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.200326 6513 raft_consensus.cc:374] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.200613 6513 raft_consensus.cc:729] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.201478 6758 leader_election.cc:288] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:04.201217 6513 consensus_queue.cc:260] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.187892 6514 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:04.178026 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1658a580bb164365a9df9fed7a641de0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.204897 6469 raft_consensus.cc:2379] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:04.188674 6364 leader_election.cc:302] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.207309 6728 raft_consensus.cc:2781] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.207866 6728 raft_consensus.cc:482] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.208334 6728 raft_consensus.cc:3037] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.211863 6513 ts_tablet_manager.cc:1398] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.060s user 0.002s sys 0.006s
I20240503 14:05:04.189213 6642 leader_election.cc:288] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.192596 6435 leader_election.cc:302] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.195626 6436 leader_election.cc:302] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.212904 6513 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:04.214807 6679 raft_consensus.cc:2781] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.215480 6679 raft_consensus.cc:686] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:04.215575 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:04.197502 6397 raft_consensus.cc:2454] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:04.216746 6724 raft_consensus.cc:2726] T 1658a580bb164365a9df9fed7a641de0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.216596 6679 consensus_queue.cc:237] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.219210 6728 raft_consensus.cc:504] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.221029 6289 leader_election.cc:302] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:04.221411 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.221729 6551 raft_consensus.cc:2781] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.222187 6551 raft_consensus.cc:686] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
W20240503 14:05:04.223071 6436 leader_election.cc:341] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.222952 6551 consensus_queue.cc:237] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:04.225996 6435 leader_election.cc:341] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.226519 6435 leader_election.cc:302] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.227128 6772 raft_consensus.cc:2726] T 1658a580bb164365a9df9fed7a641de0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.227283 6727 raft_consensus.cc:2726] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.227474 6671 raft_consensus.cc:482] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.228013 6671 raft_consensus.cc:504] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.230736 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:04.231223 6671 leader_election.cc:288] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.231402 6469 raft_consensus.cc:3037] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.233745 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:04.237653 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.238507 6471 raft_consensus.cc:2454] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
W20240503 14:05:04.240900 6362 leader_election.cc:341] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.242406 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:04.243124 6397 raft_consensus.cc:2454] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:04.245139 6286 leader_election.cc:302] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.248734 6772 raft_consensus.cc:2781] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.249176 6772 raft_consensus.cc:482] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.249523 6772 raft_consensus.cc:3037] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.269210 6469 raft_consensus.cc:2454] T 41d1828e46eb4868900cf3ec7e69b5b0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:04.270512 6364 leader_election.cc:302] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.271579 6642 raft_consensus.cc:2781] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.272132 6642 raft_consensus.cc:686] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:04.273020 6642 consensus_queue.cc:237] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.268302 6772 raft_consensus.cc:504] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.279480 6728 leader_election.cc:288] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.282851 6514 tablet_bootstrap.cc:654] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.286585 6513 tablet_bootstrap.cc:654] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.295163 6728 raft_consensus.cc:482] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.295764 6728 raft_consensus.cc:504] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.296000 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:04.297973 6397 raft_consensus.cc:3037] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.299639 6728 leader_election.cc:288] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.308538 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.309356 6469 raft_consensus.cc:2454] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:04.313421 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a594e34044584aa09a50d15954cbb81b" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:04.314114 6469 raft_consensus.cc:3037] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.315109 6512 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.315609 6512 ts_tablet_manager.cc:1367] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.208s user 0.029s sys 0.025s
I20240503 14:05:04.308573 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.317725 6323 raft_consensus.cc:2454] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:04.318277 6512 raft_consensus.cc:348] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.319025 6512 raft_consensus.cc:374] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.319304 6512 raft_consensus.cc:729] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.320951 6362 leader_election.cc:302] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.319924 6512 consensus_queue.cc:260] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.323771 6512 ts_tablet_manager.cc:1398] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:05:04.323973 6568 raft_consensus.cc:2781] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.324399 6568 raft_consensus.cc:482] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.324785 6512 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:04.324784 6568 raft_consensus.cc:3037] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.312173 6772 leader_election.cc:288] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.332551 6679 consensus_queue.cc:1035] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.339275 6221 catalog_manager.cc:5484] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:04.344556 6221 catalog_manager.cc:5484] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:04.365365 6218 catalog_manager.cc:5484] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:04.399919 6512 tablet_bootstrap.cc:654] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.353797 6568 raft_consensus.cc:504] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.409361 6568 leader_election.cc:288] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.411063 6397 raft_consensus.cc:2454] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:04.414019 6286 leader_election.cc:302] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.414078 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:04.415115 6323 raft_consensus.cc:3037] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.415246 6772 raft_consensus.cc:2781] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
W20240503 14:05:04.400513 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 74226.9us
I20240503 14:05:04.417240 6772 raft_consensus.cc:686] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:04.418896 6772 consensus_queue.cc:237] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.423498 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e09030e41364fbaadad3e4ea90d6166" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:04.424268 6471 raft_consensus.cc:3037] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.424780 6469 raft_consensus.cc:2454] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:04.425822 6514 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:04.426342 6514 ts_tablet_manager.cc:1367] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.239s user 0.005s sys 0.011s
I20240503 14:05:04.431531 6513 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:04.429790 6514 raft_consensus.cc:348] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.440338 6514 raft_consensus.cc:374] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.440593 6513 ts_tablet_manager.cc:1367] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.228s user 0.005s sys 0.015s
I20240503 14:05:04.440692 6514 raft_consensus.cc:729] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.437947 6323 raft_consensus.cc:2454] T 2e09030e41364fbaadad3e4ea90d6166 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:04.443665 6514 consensus_queue.cc:260] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.446092 6514 ts_tablet_manager.cc:1398] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20240503 14:05:04.445719 6513 raft_consensus.cc:348] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.446650 6513 raft_consensus.cc:374] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.446997 6513 raft_consensus.cc:729] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.447824 6513 consensus_queue.cc:260] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.450284 6362 leader_election.cc:302] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.451243 6724 raft_consensus.cc:2781] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.452044 6514 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:04.451870 6724 raft_consensus.cc:686] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:04.453956 6724 consensus_queue.cc:237] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.468063 6513 ts_tablet_manager.cc:1398] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20240503 14:05:04.471824 6513 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:04.473874 6471 raft_consensus.cc:2454] T 2e09030e41364fbaadad3e4ea90d6166 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:04.487772 6758 consensus_queue.cc:1035] T 1fd4c415b34640bd82515afe5d3a7632 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.502841 6514 tablet_bootstrap.cc:654] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.511126 6512 tablet_bootstrap.cc:492] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.511869 6512 ts_tablet_manager.cc:1367] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.187s user 0.035s sys 0.011s
I20240503 14:05:04.516839 6221 catalog_manager.cc:5484] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:04.528764 6512 raft_consensus.cc:348] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.530289 6512 raft_consensus.cc:374] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.532446 6512 raft_consensus.cc:729] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.533238 6512 consensus_queue.cc:260] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.535657 6512 ts_tablet_manager.cc:1398] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20240503 14:05:04.536723 6512 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:04.537734 6759 consensus_queue.cc:1035] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.532027 6819 raft_consensus.cc:482] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.539574 6819 raft_consensus.cc:504] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.548717 6819 leader_election.cc:288] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.543212 6221 catalog_manager.cc:5484] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:04.553493 6512 tablet_bootstrap.cc:654] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:04.556234 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:04.557914 6513 tablet_bootstrap.cc:654] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.562630 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.567764 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:04.568572 6435 consensus_peers.cc:485] T e09711da83964cb1b409403e16852e17 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Couldn't send request to peer 90a66da3b59e4511bdc1d35930fe558d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:05:04.569805 6364 leader_election.cc:341] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:04.570461 6362 leader_election.cc:341] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.570866 6362 leader_election.cc:302] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.577838 6820 raft_consensus.cc:2726] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.586485 6514 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:04.587086 6514 ts_tablet_manager.cc:1367] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.140s user 0.019s sys 0.002s
I20240503 14:05:04.597841 6512 tablet_bootstrap.cc:492] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.600029 6512 ts_tablet_manager.cc:1367] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.000s
I20240503 14:05:04.599020 6514 raft_consensus.cc:348] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.602380 6514 raft_consensus.cc:374] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.602761 6514 raft_consensus.cc:729] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.603637 6514 consensus_queue.cc:260] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.605506 6512 raft_consensus.cc:348] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.609177 6512 raft_consensus.cc:374] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.609573 6512 raft_consensus.cc:729] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.610278 6512 consensus_queue.cc:260] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.607976 6514 ts_tablet_manager.cc:1398] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.021s user 0.007s sys 0.002s
W20240503 14:05:04.617533 6436 consensus_peers.cc:485] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Couldn't send request to peer 9a9ba562b78c4746b83a40ca7c30c00d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:04.618988 6514 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:04.626045 6514 tablet_bootstrap.cc:654] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.633749 6512 ts_tablet_manager.cc:1398] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20240503 14:05:04.635607 6512 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:04.642186 6512 tablet_bootstrap.cc:654] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.652667 6589 raft_consensus.cc:482] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.653321 6589 raft_consensus.cc:504] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.657655 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:04.658533 6396 raft_consensus.cc:2454] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:04.655951 6589 leader_election.cc:288] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.660199 6286 leader_election.cc:302] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.663120 6589 raft_consensus.cc:2781] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:04.662036 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.664767 6589 raft_consensus.cc:482] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:04.665298 6589 raft_consensus.cc:3037] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.664963 6469 raft_consensus.cc:2454] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:04.668560 6514 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:04.675057 6727 consensus_queue.cc:1035] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.698478 6551 consensus_queue.cc:1035] T 1658a580bb164365a9df9fed7a641de0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:04.716553 6772 raft_consensus.cc:482] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.717175 6772 raft_consensus.cc:504] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.719684 6589 raft_consensus.cc:504] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.721129 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:04.723012 6286 leader_election.cc:341] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.719686 6772 leader_election.cc:288] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.728153 6322 raft_consensus.cc:3037] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.730918 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.730803 6589 leader_election.cc:288] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:04.733824 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:04.734699 6397 raft_consensus.cc:3037] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.737684 6512 tablet_bootstrap.cc:492] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.738225 6512 ts_tablet_manager.cc:1367] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.103s user 0.037s sys 0.022s
I20240503 14:05:04.739387 6727 consensus_queue.cc:1035] T c9b840e10839487fac480c2bbf79f329 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.743872 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f627e726d194488cb11419bbff15aa2d" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:04.744686 6471 raft_consensus.cc:3037] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.747025 6512 raft_consensus.cc:348] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.747812 6512 raft_consensus.cc:374] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.748128 6512 raft_consensus.cc:729] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
W20240503 14:05:04.754252 6289 leader_election.cc:341] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.763054 6512 consensus_queue.cc:260] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.763303 6289 leader_election.cc:302] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:04.765648 6671 raft_consensus.cc:2726] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.767972 6318 raft_consensus.cc:3037] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:04.770848 6512 ts_tablet_manager.cc:1398] T 5f401553cb154845b5d0a890810a9837 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20240503 14:05:04.771411 6471 raft_consensus.cc:2454] T f627e726d194488cb11419bbff15aa2d P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:04.771914 6512 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:04.762239 6514 ts_tablet_manager.cc:1367] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.143s user 0.008s sys 0.006s
I20240503 14:05:04.775791 6397 raft_consensus.cc:2454] T f627e726d194488cb11419bbff15aa2d P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:04.810541 6513 tablet_bootstrap.cc:492] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:04.840615 6513 ts_tablet_manager.cc:1367] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.369s user 0.080s sys 0.080s
I20240503 14:05:04.799769 6819 consensus_queue.cc:1035] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20240503 14:05:04.842092 6512 tablet_bootstrap.cc:654] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:04.843693 6513 raft_consensus.cc:348] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.844589 6513 raft_consensus.cc:374] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.844890 6513 raft_consensus.cc:729] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.843613 6514 raft_consensus.cc:348] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.846253 6514 raft_consensus.cc:374] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.846547 6514 raft_consensus.cc:729] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.885275 6286 leader_election.cc:302] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:04.917390 6671 raft_consensus.cc:2781] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:04.845650 6513 consensus_queue.cc:260] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.922459 6513 ts_tablet_manager.cc:1398] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.081s user 0.009s sys 0.000s
I20240503 14:05:04.909806 6514 consensus_queue.cc:260] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.938336 6512 tablet_bootstrap.cc:492] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:04.939267 6512 ts_tablet_manager.cc:1367] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.168s user 0.022s sys 0.021s
I20240503 14:05:04.945196 6759 raft_consensus.cc:482] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.947048 6671 raft_consensus.cc:686] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:04.948572 6671 consensus_queue.cc:237] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.956964 6759 raft_consensus.cc:504] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.961143 6513 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:04.962455 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:04.964412 6724 raft_consensus.cc:482] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:04.964680 6819 consensus_queue.cc:1035] T 41d1828e46eb4868900cf3ec7e69b5b0 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:04.965170 6724 raft_consensus.cc:504] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:04.965516 6436 leader_election.cc:341] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.969296 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:04.973851 6435 leader_election.cc:341] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:04.974488 6435 leader_election.cc:302] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:04.959882 6759 leader_election.cc:288] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:04.971418 6512 raft_consensus.cc:348] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.978562 6512 raft_consensus.cc:374] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:04.982967 6512 raft_consensus.cc:729] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:04.985124 6512 consensus_queue.cc:260] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:04.991488 6727 raft_consensus.cc:2726] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:04.992496 6514 ts_tablet_manager.cc:1398] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.155s user 0.010s sys 0.005s
I20240503 14:05:05.000480 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:04.988685 6724 leader_election.cc:288] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.000816 6514 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:05.007418 6513 tablet_bootstrap.cc:654] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:04.991478 6289 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 70593.5us
I20240503 14:05:05.010493 6819 consensus_queue.cc:1035] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:05.019043 6772 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:05.024781 6512 ts_tablet_manager.cc:1398] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.085s user 0.002s sys 0.010s
I20240503 14:05:05.026604 6512 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:05.037899 6843 raft_consensus.cc:482] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.038800 6843 raft_consensus.cc:504] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.041638 6843 leader_election.cc:288] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.048950 6514 tablet_bootstrap.cc:654] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.055872 6512 tablet_bootstrap.cc:654] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.060235 6513 tablet_bootstrap.cc:492] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:05.061182 6513 ts_tablet_manager.cc:1367] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.129s user 0.007s sys 0.008s
I20240503 14:05:05.075031 6513 raft_consensus.cc:348] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.077133 6513 raft_consensus.cc:374] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.077487 6513 raft_consensus.cc:729] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.079560 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09711da83964cb1b409403e16852e17" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:05.081565 6513 consensus_queue.cc:260] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.154578 6513 ts_tablet_manager.cc:1398] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.093s user 0.004s sys 0.003s
I20240503 14:05:05.155822 6513 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:05.212680 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:05.240870 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 88812.1us
I20240503 14:05:05.247579 6362 leader_election.cc:302] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:05.252704 6642 raft_consensus.cc:2726] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:05.256950 6218 catalog_manager.cc:5484] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:05.257828 6820 consensus_queue.cc:1035] T 2e09030e41364fbaadad3e4ea90d6166 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:05.260812 6319 raft_consensus.cc:3037] T c9b840e10839487fac480c2bbf79f329 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:05.273049 6435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 112398us
I20240503 14:05:05.291870 6398 raft_consensus.cc:3037] T e09711da83964cb1b409403e16852e17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.295511 6514 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:05.296415 6514 ts_tablet_manager.cc:1367] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.296s user 0.015s sys 0.006s
I20240503 14:05:05.298846 6513 tablet_bootstrap.cc:654] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.291971 6671 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240503 14:05:05.317250 6551 raft_consensus.cc:482] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.317840 6551 raft_consensus.cc:504] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.330335 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d1828e46eb4868900cf3ec7e69b5b0" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:05.335757 6551 leader_election.cc:288] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.355055 6289 leader_election.cc:302] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:05.365358 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:05.366202 6396 raft_consensus.cc:2454] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.358649 6772 raft_consensus.cc:2726] T 41d1828e46eb4868900cf3ec7e69b5b0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:05:05.371330 6512 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:05.371897 6512 ts_tablet_manager.cc:1367] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.346s user 0.014s sys 0.019s
I20240503 14:05:05.374578 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:05.360486 6514 raft_consensus.cc:348] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.376891 6514 raft_consensus.cc:374] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.377264 6514 raft_consensus.cc:729] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.375922 6435 leader_election.cc:302] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:05.380422 6322 raft_consensus.cc:2454] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.383493 6514 consensus_queue.cc:260] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.387887 6852 raft_consensus.cc:2781] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:05.388406 6852 raft_consensus.cc:482] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:05.388779 6852 raft_consensus.cc:3037] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.397434 6512 raft_consensus.cc:348] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.398399 6512 raft_consensus.cc:374] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.398720 6512 raft_consensus.cc:729] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.399336 6512 consensus_queue.cc:260] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.402312 6512 ts_tablet_manager.cc:1398] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.026s user 0.004s sys 0.001s
I20240503 14:05:05.404556 6512 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
W20240503 14:05:05.413327 6289 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 55179us
I20240503 14:05:05.460755 6860 raft_consensus.cc:482] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.461341 6860 raft_consensus.cc:504] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.464440 6860 leader_election.cc:288] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.464648 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:05.466490 6286 leader_election.cc:341] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:05.470547 6514 ts_tablet_manager.cc:1398] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.174s user 0.008s sys 0.000s
I20240503 14:05:05.471379 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:05.471798 6514 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
W20240503 14:05:05.475198 6289 leader_election.cc:341] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:05.476038 6512 tablet_bootstrap.cc:654] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.475680 6289 leader_election.cc:302] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:05.478229 6860 raft_consensus.cc:2726] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:05.469962 6852 raft_consensus.cc:504] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.485057 6513 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:05.485544 6513 ts_tablet_manager.cc:1367] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.330s user 0.014s sys 0.011s
I20240503 14:05:05.486388 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:05.487090 6320 raft_consensus.cc:3037] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.487562 6852 leader_election.cc:288] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.489079 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2af980c77644cb8af111aa4f519f173" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:05.489804 6398 raft_consensus.cc:3037] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.519831 6320 raft_consensus.cc:2454] T a2af980c77644cb8af111aa4f519f173 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:05.520972 6436 leader_election.cc:302] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.521694 6727 raft_consensus.cc:2781] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:05.522207 6727 raft_consensus.cc:686] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:05.520404 6398 raft_consensus.cc:2454] T a2af980c77644cb8af111aa4f519f173 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:05.523073 6727 consensus_queue.cc:237] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.523612 6671 consensus_queue.cc:1035] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:05.535567 6513 raft_consensus.cc:348] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.536351 6513 raft_consensus.cc:374] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.536638 6513 raft_consensus.cc:729] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.538039 6513 consensus_queue.cc:260] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.539676 6513 ts_tablet_manager.cc:1398] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20240503 14:05:05.541087 6513 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:05.551009 6513 tablet_bootstrap.cc:654] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.554488 6512 tablet_bootstrap.cc:492] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:05.555042 6512 ts_tablet_manager.cc:1367] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.152s user 0.022s sys 0.009s
I20240503 14:05:05.564493 6512 raft_consensus.cc:348] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.565274 6512 raft_consensus.cc:374] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.565542 6512 raft_consensus.cc:729] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.566172 6512 consensus_queue.cc:260] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.566907 6860 consensus_queue.cc:1035] T f627e726d194488cb11419bbff15aa2d P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:05.570984 6218 catalog_manager.cc:5484] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:05.583698 6512 ts_tablet_manager.cc:1398] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240503 14:05:05.587838 6514 tablet_bootstrap.cc:654] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.588505 6512 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:05.605831 6512 tablet_bootstrap.cc:654] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.605967 6513 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:05.614874 6514 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:05.628422 6514 ts_tablet_manager.cc:1367] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.157s user 0.006s sys 0.007s
I20240503 14:05:05.628378 6513 ts_tablet_manager.cc:1367] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.004s
I20240503 14:05:05.631359 6514 raft_consensus.cc:348] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.632189 6514 raft_consensus.cc:374] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.632481 6514 raft_consensus.cc:729] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
W20240503 14:05:05.632846 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:05.631848 6513 raft_consensus.cc:348] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.633926 6513 raft_consensus.cc:374] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.634228 6513 raft_consensus.cc:729] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.634918 6513 consensus_queue.cc:260] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.633219 6514 consensus_queue.cc:260] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.637962 6512 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:05.638463 6512 ts_tablet_manager.cc:1367] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.001s
I20240503 14:05:05.641268 6512 raft_consensus.cc:348] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.642021 6512 raft_consensus.cc:374] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.642308 6512 raft_consensus.cc:729] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.643070 6512 consensus_queue.cc:260] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.644814 6514 ts_tablet_manager.cc:1398] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240503 14:05:05.645465 6512 ts_tablet_manager.cc:1398] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240503 14:05:05.645692 6513 ts_tablet_manager.cc:1398] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.017s user 0.002s sys 0.005s
I20240503 14:05:05.646508 6512 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:05.646615 6513 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:05.649586 6514 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:05.660205 6514 tablet_bootstrap.cc:654] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.678778 6513 tablet_bootstrap.cc:654] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.682240 6512 tablet_bootstrap.cc:654] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.704304 6860 raft_consensus.cc:482] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.705124 6860 raft_consensus.cc:504] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.708890 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:05.709697 6398 raft_consensus.cc:2454] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:05.712230 6286 leader_election.cc:302] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.714092 6671 raft_consensus.cc:2781] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:05.714498 6671 raft_consensus.cc:482] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:05.714849 6671 raft_consensus.cc:3037] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.726868 6514 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:05.733995 6860 leader_election.cc:288] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.735956 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:05:05.738004 6289 leader_election.cc:341] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:05.739266 6512 tablet_bootstrap.cc:492] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:05.739820 6512 ts_tablet_manager.cc:1367] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.094s user 0.008s sys 0.004s
I20240503 14:05:05.742401 6512 raft_consensus.cc:348] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.743153 6512 raft_consensus.cc:374] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.743470 6512 raft_consensus.cc:729] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.739187 6514 ts_tablet_manager.cc:1367] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.020s
I20240503 14:05:05.749208 6727 raft_consensus.cc:482] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.749764 6727 raft_consensus.cc:504] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.748126 6512 consensus_queue.cc:260] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.755441 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:05.756624 6513 tablet_bootstrap.cc:492] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:05.757243 6513 ts_tablet_manager.cc:1367] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.111s user 0.023s sys 0.022s
I20240503 14:05:05.758927 6671 raft_consensus.cc:504] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.763501 6727 leader_election.cc:288] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.764616 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:05.764966 6320 raft_consensus.cc:2454] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.765522 6395 raft_consensus.cc:3037] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.769778 6512 ts_tablet_manager.cc:1398] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20240503 14:05:05.764513 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:05.770936 6398 raft_consensus.cc:2454] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.771878 6436 leader_election.cc:302] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.775753 6727 raft_consensus.cc:2781] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:05.776232 6727 raft_consensus.cc:482] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:05.776594 6727 raft_consensus.cc:3037] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.777878 6395 raft_consensus.cc:2454] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:05.778362 6512 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:05.779422 6286 leader_election.cc:302] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.780849 6860 raft_consensus.cc:2781] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:05.781347 6860 raft_consensus.cc:686] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:05.782220 6860 consensus_queue.cc:237] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.784484 6727 raft_consensus.cc:504] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.787670 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:05.788419 6398 raft_consensus.cc:3037] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.747630 6514 raft_consensus.cc:348] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.787849 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5da6c0c9a9554a998f3efbacf240ca17" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:05.767339 6513 raft_consensus.cc:348] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.793105 6513 raft_consensus.cc:374] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.793159 6320 raft_consensus.cc:3037] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.793504 6513 raft_consensus.cc:729] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.794196 6513 consensus_queue.cc:260] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.786660 6727 leader_election.cc:288] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.794677 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:05.795727 6514 raft_consensus.cc:374] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.796118 6514 raft_consensus.cc:729] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.798202 6514 consensus_queue.cc:260] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.799418 6513 ts_tablet_manager.cc:1398] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20240503 14:05:05.792245 6671 leader_election.cc:288] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.800738 6513 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:05.802937 6320 raft_consensus.cc:2454] T 5da6c0c9a9554a998f3efbacf240ca17 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:05.804822 6398 raft_consensus.cc:2454] T 5da6c0c9a9554a998f3efbacf240ca17 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:05.830231 6512 tablet_bootstrap.cc:654] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.837435 6435 leader_election.cc:302] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
W20240503 14:05:05.838781 6289 leader_election.cc:341] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:05.850062 6727 raft_consensus.cc:2781] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:05.851847 6727 raft_consensus.cc:686] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:05.854193 6728 raft_consensus.cc:482] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.853997 6727 consensus_queue.cc:237] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.818445 6514 ts_tablet_manager.cc:1398] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.074s user 0.008s sys 0.000s
I20240503 14:05:05.854807 6728 raft_consensus.cc:504] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.856109 6514 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:05.856938 6728 leader_election.cc:288] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.858417 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:05.859776 6320 raft_consensus.cc:2454] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:05.818895 6852 raft_consensus.cc:482] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:05.860401 6852 raft_consensus.cc:504] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.861081 6362 leader_election.cc:302] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.861860 6728 raft_consensus.cc:2781] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:05.862232 6728 raft_consensus.cc:482] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:05.862432 6852 leader_election.cc:288] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.862557 6728 raft_consensus.cc:3037] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.864305 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:05.865092 6468 raft_consensus.cc:2454] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:05.869947 6514 tablet_bootstrap.cc:654] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.871500 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:05.872634 6320 raft_consensus.cc:2454] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.873883 6436 leader_election.cc:302] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.880507 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:05.880867 6758 raft_consensus.cc:2781] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:05.881290 6758 raft_consensus.cc:482] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:05.881306 6398 raft_consensus.cc:2454] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:05.881767 6758 raft_consensus.cc:3037] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.814445 6513 tablet_bootstrap.cc:654] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:05.891522 6758 raft_consensus.cc:504] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.898157 6728 raft_consensus.cc:504] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.907480 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:05.908195 6319 raft_consensus.cc:3037] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.910634 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:05.911389 6468 raft_consensus.cc:3037] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.912942 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8587555ef5ac4e31b7ab986f265fd3d7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:05.913630 6321 raft_consensus.cc:3037] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.919124 6468 raft_consensus.cc:2454] T 8587555ef5ac4e31b7ab986f265fd3d7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:05.920634 6364 leader_election.cc:302] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:05.921324 6319 raft_consensus.cc:2454] T 6741d371ee4e47b39f1c444c9fb8fe8a P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:05.921504 6724 raft_consensus.cc:2781] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:05.922067 6724 raft_consensus.cc:686] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:05.922899 6724 consensus_queue.cc:237] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.923483 6321 raft_consensus.cc:2454] T 8587555ef5ac4e31b7ab986f265fd3d7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:05.927394 6218 catalog_manager.cc:5484] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:05.930958 6728 leader_election.cc:288] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:05.939170 6436 leader_election.cc:302] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:05.940107 6512 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:05.940630 6512 ts_tablet_manager.cc:1367] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.163s user 0.038s sys 0.039s
I20240503 14:05:05.940713 6852 raft_consensus.cc:2781] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:05.943027 6852 raft_consensus.cc:686] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:05.943826 6758 leader_election.cc:288] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:05.944173 6852 consensus_queue.cc:237] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.947712 6851 consensus_queue.cc:1035] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:05.951406 6512 raft_consensus.cc:348] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.952235 6512 raft_consensus.cc:374] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:05.952525 6512 raft_consensus.cc:729] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:05.953219 6512 consensus_queue.cc:260] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:05.955478 6512 ts_tablet_manager.cc:1398] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240503 14:05:05.956609 6512 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:05.964049 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6741d371ee4e47b39f1c444c9fb8fe8a" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:05.964936 6395 raft_consensus.cc:3037] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:05.974457 6395 raft_consensus.cc:2454] T 6741d371ee4e47b39f1c444c9fb8fe8a P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:06.005789 6512 tablet_bootstrap.cc:654] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.010483 6514 tablet_bootstrap.cc:492] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:06.011076 6514 ts_tablet_manager.cc:1367] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.155s user 0.017s sys 0.000s
I20240503 14:05:06.011729 6513 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:06.012224 6513 ts_tablet_manager.cc:1367] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.212s user 0.011s sys 0.009s
I20240503 14:05:06.024780 6218 catalog_manager.cc:5484] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:06.034041 6551 consensus_queue.cc:1035] T a2af980c77644cb8af111aa4f519f173 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:06.036890 6514 raft_consensus.cc:348] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.037775 6514 raft_consensus.cc:374] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.038079 6514 raft_consensus.cc:729] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.038774 6514 consensus_queue.cc:260] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.051290 6514 ts_tablet_manager.cc:1398] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.040s user 0.006s sys 0.001s
I20240503 14:05:06.052426 6514 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:06.052831 6513 raft_consensus.cc:348] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.053658 6513 raft_consensus.cc:374] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.053929 6513 raft_consensus.cc:729] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.054924 6513 consensus_queue.cc:260] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.059114 6220 catalog_manager.cc:5484] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:06.073340 6513 ts_tablet_manager.cc:1398] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20240503 14:05:06.075671 6513 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:06.086946 6514 tablet_bootstrap.cc:654] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.105748 6221 catalog_manager.cc:5484] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:06.117498 6513 tablet_bootstrap.cc:654] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.122678 6512 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:06.123374 6512 ts_tablet_manager.cc:1367] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.167s user 0.014s sys 0.012s
I20240503 14:05:06.125916 6512 raft_consensus.cc:348] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.139137 6512 raft_consensus.cc:374] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.139483 6512 raft_consensus.cc:729] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.140355 6512 consensus_queue.cc:260] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.140952 6513 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:06.141297 6514 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:06.141484 6513 ts_tablet_manager.cc:1367] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.005s
I20240503 14:05:06.141801 6514 ts_tablet_manager.cc:1367] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.001s
I20240503 14:05:06.144274 6513 raft_consensus.cc:348] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.145092 6513 raft_consensus.cc:374] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.145385 6513 raft_consensus.cc:729] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.147933 6514 raft_consensus.cc:348] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.148793 6514 raft_consensus.cc:374] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.149072 6514 raft_consensus.cc:729] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.149744 6513 consensus_queue.cc:260] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.152537 6513 ts_tablet_manager.cc:1398] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240503 14:05:06.153541 6513 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:06.155970 6514 consensus_queue.cc:260] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.158640 6514 ts_tablet_manager.cc:1398] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240503 14:05:06.159746 6514 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:06.169090 6513 tablet_bootstrap.cc:654] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.174518 6514 tablet_bootstrap.cc:654] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.181988 6724 raft_consensus.cc:482] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.182512 6724 raft_consensus.cc:504] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.190002 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.190752 6320 raft_consensus.cc:2454] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:06.192054 6362 leader_election.cc:302] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:06.193040 6728 raft_consensus.cc:2781] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.193481 6728 raft_consensus.cc:482] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.193852 6728 raft_consensus.cc:3037] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.198143 6727 raft_consensus.cc:482] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.198833 6727 raft_consensus.cc:504] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.201017 6727 leader_election.cc:288] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.202374 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:06.204535 6435 leader_election.cc:341] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.216698 6724 leader_election.cc:288] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.218932 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.220077 6471 raft_consensus.cc:2454] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:06.220489 6727 raft_consensus.cc:482] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.221037 6727 raft_consensus.cc:504] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.223233 6727 leader_election.cc:288] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.224480 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.225391 6397 raft_consensus.cc:2379] T 88ac2eddffd74f50bc6b87f3a7c0827c P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:06.226713 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
W20240503 14:05:06.228663 6436 leader_election.cc:341] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.229215 6436 leader_election.cc:302] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.231942 6758 raft_consensus.cc:2726] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.233863 6320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ac2eddffd74f50bc6b87f3a7c0827c" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.236778 6436 leader_election.cc:302] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.237826 6727 raft_consensus.cc:3037] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.181527 6512 ts_tablet_manager.cc:1398] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20240503 14:05:06.247936 6512 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:06.248706 6728 raft_consensus.cc:504] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.251525 6727 raft_consensus.cc:2726] T 88ac2eddffd74f50bc6b87f3a7c0827c P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.253158 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:06.256703 6319 raft_consensus.cc:3037] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.264676 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3586823a1bd546f08b8007d4b55ef43e" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:06.265841 6469 raft_consensus.cc:3037] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.266312 6319 raft_consensus.cc:2454] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:06.267975 6362 leader_election.cc:302] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:06.269037 6724 raft_consensus.cc:2781] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:06.269483 6728 leader_election.cc:288] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.269522 6724 raft_consensus.cc:686] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:06.272931 6724 consensus_queue.cc:237] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.284489 6512 tablet_bootstrap.cc:654] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.295127 6772 consensus_queue.cc:1035] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.292848 6514 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:06.284529 6513 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:06.297456 6513 ts_tablet_manager.cc:1367] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.144s user 0.018s sys 0.034s
I20240503 14:05:06.300668 6514 ts_tablet_manager.cc:1367] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.141s user 0.036s sys 0.016s
I20240503 14:05:06.303474 6514 raft_consensus.cc:348] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.296674 6469 raft_consensus.cc:2454] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:06.300276 6513 raft_consensus.cc:348] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.305941 6513 raft_consensus.cc:374] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.306282 6513 raft_consensus.cc:729] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.306994 6513 consensus_queue.cc:260] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.304311 6514 raft_consensus.cc:374] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.308066 6514 raft_consensus.cc:729] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.308918 6514 consensus_queue.cc:260] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.316154 6514 ts_tablet_manager.cc:1398] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240503 14:05:06.317237 6514 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:06.325791 6513 ts_tablet_manager.cc:1398] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240503 14:05:06.326946 6513 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:06.331439 6221 catalog_manager.cc:5484] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:06.333063 6860 consensus_queue.cc:1035] T 88ac2eddffd74f50bc6b87f3a7c0827c P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.472640 6513 tablet_bootstrap.cc:700] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: Time spent opening tablet: real 0.129s user 0.000s sys 0.010s
I20240503 14:05:06.473513 6896 raft_consensus.cc:482] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.474196 6896 raft_consensus.cc:504] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.476625 6896 leader_election.cc:288] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.485848 6512 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:06.490711 6512 ts_tablet_manager.cc:1367] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.251s user 0.029s sys 0.013s
I20240503 14:05:06.491060 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.490039 6514 tablet_bootstrap.cc:700] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent opening tablet: real 0.145s user 0.009s sys 0.001s
I20240503 14:05:06.493675 6514 tablet_bootstrap.cc:654] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.473551 6513 tablet_bootstrap.cc:654] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:06.502918 6362 leader_election.cc:341] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.503497 6727 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.504038 6727 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.505973 6727 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.506328 6671 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.506846 6671 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.511745 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.512533 6512 raft_consensus.cc:348] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:06.513338 6286 leader_election.cc:341] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.517266 6671 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.513353 6512 raft_consensus.cc:374] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.518260 6512 raft_consensus.cc:729] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.518911 6512 consensus_queue.cc:260] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.520625 6512 ts_tablet_manager.cc:1398] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20240503 14:05:06.528384 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.537794 6513 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:06.535461 6759 consensus_queue.cc:1035] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.538427 6513 ts_tablet_manager.cc:1367] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.212s user 0.006s sys 0.012s
I20240503 14:05:06.541882 6512 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:06.544348 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.545162 6316 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:06.551134 6514 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:06.551704 6514 ts_tablet_manager.cc:1367] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.235s user 0.017s sys 0.002s
W20240503 14:05:06.552520 6364 leader_election.cc:341] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.553015 6364 leader_election.cc:302] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.553534 6512 tablet_bootstrap.cc:654] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.554242 6820 raft_consensus.cc:2726] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.554383 6514 raft_consensus.cc:348] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.555248 6514 raft_consensus.cc:374] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.555552 6514 raft_consensus.cc:729] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.560031 6899 consensus_queue.cc:1035] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.560631 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.561376 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.563714 6468 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:06.567601 6513 raft_consensus.cc:348] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.568444 6513 raft_consensus.cc:374] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.568734 6513 raft_consensus.cc:729] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.573551 6727 consensus_queue.cc:1035] T 5da6c0c9a9554a998f3efbacf240ca17 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:05:06.577582 6435 leader_election.cc:341] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.578992 6758 consensus_queue.cc:1035] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:06.583849 6512 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:06.584491 6512 ts_tablet_manager.cc:1367] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.007s
I20240503 14:05:06.594420 6436 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:06.595770 6901 raft_consensus.cc:2781] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.595562 6899 consensus_queue.cc:1035] T 8587555ef5ac4e31b7ab986f265fd3d7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.596261 6901 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.596650 6901 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.597476 6512 raft_consensus.cc:348] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.598275 6512 raft_consensus.cc:374] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.598567 6512 raft_consensus.cc:729] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.600034 6512 consensus_queue.cc:260] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.601840 6512 ts_tablet_manager.cc:1398] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.017s user 0.001s sys 0.005s
I20240503 14:05:06.602902 6512 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:06.608333 6514 consensus_queue.cc:260] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.609412 6898 raft_consensus.cc:482] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.609985 6898 raft_consensus.cc:504] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.611373 6512 tablet_bootstrap.cc:654] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.614161 6898 leader_election.cc:288] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.622228 6513 consensus_queue.cc:260] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.624672 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1407f54e751147b3b05f2b4083dbc041" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.627087 6851 consensus_queue.cc:1035] T 6741d371ee4e47b39f1c444c9fb8fe8a P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:06.634308 6514 ts_tablet_manager.cc:1398] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.082s user 0.008s sys 0.000s
I20240503 14:05:06.635601 6289 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
W20240503 14:05:06.641216 6286 leader_election.cc:341] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.640329 6513 ts_tablet_manager.cc:1398] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.102s user 0.010s sys 0.000s
I20240503 14:05:06.647341 6514 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:06.644804 6898 raft_consensus.cc:2781] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.649549 6898 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.649955 6898 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.650218 6852 raft_consensus.cc:482] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.653482 6852 raft_consensus.cc:504] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.654414 6513 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:06.655335 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1407f54e751147b3b05f2b4083dbc041" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.655718 6852 leader_election.cc:288] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.656282 6468 raft_consensus.cc:2454] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:06.659911 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.677594 6724 raft_consensus.cc:482] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.698540 6724 raft_consensus.cc:504] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.718331 6724 leader_election.cc:288] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.730057 6898 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.657276 6901 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.739125 6468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.739979 6289 leader_election.cc:302] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:06.742199 6671 raft_consensus.cc:2781] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.742622 6671 raft_consensus.cc:482] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.743021 6671 raft_consensus.cc:3037] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:06.739207 6436 leader_election.cc:341] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.750242 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.751034 6321 raft_consensus.cc:2454] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
I20240503 14:05:06.752334 6513 tablet_bootstrap.cc:654] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.754890 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:06.758579 6898 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.761955 6671 raft_consensus.cc:504] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.769516 6512 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:06.770038 6901 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.770128 6512 ts_tablet_manager.cc:1367] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.004s
I20240503 14:05:06.771194 6362 leader_election.cc:302] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:06.775115 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:06.775338 6724 raft_consensus.cc:2781] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.775846 6724 raft_consensus.cc:482] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.776218 6724 raft_consensus.cc:3037] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.779186 6471 raft_consensus.cc:2379] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:06.786887 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1407f54e751147b3b05f2b4083dbc041" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:06.787624 6470 raft_consensus.cc:3037] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.788986 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:06.789932 6316 raft_consensus.cc:2379] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:06.776239 6671 leader_election.cc:288] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.814375 6512 raft_consensus.cc:348] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.815165 6512 raft_consensus.cc:374] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.815459 6512 raft_consensus.cc:729] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.817266 6724 raft_consensus.cc:504] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.820381 6514 tablet_bootstrap.cc:654] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.821988 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:06.824959 6724 leader_election.cc:288] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
W20240503 14:05:06.825779 6435 leader_election.cc:341] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.826269 6470 raft_consensus.cc:2454] T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:06.826293 6435 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.829355 6759 raft_consensus.cc:2726] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.830544 6289 leader_election.cc:302] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:06.829135 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.831446 6671 raft_consensus.cc:2781] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:06.835126 6671 raft_consensus.cc:686] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:06.836232 6671 consensus_queue.cc:237] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:06.840397 6362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 68462.1us
I20240503 14:05:06.840777 6512 consensus_queue.cc:260] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:06.843704 6286 leader_election.cc:341] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.845026 6286 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
W20240503 14:05:06.842520 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 69969.9us
I20240503 14:05:06.847990 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:06.848423 6364 leader_election.cc:341] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.848773 6316 raft_consensus.cc:3037] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:06.855451 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:06.859823 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1407f54e751147b3b05f2b4083dbc041" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:06.860044 6860 raft_consensus.cc:2726] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.862814 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
W20240503 14:05:06.867710 6364 leader_election.cc:341] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.869367 6316 raft_consensus.cc:2454] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:06.872262 6898 raft_consensus.cc:482] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.872797 6898 raft_consensus.cc:504] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.874943 6898 leader_election.cc:288] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.875612 6513 tablet_bootstrap.cc:492] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:06.876436 6512 ts_tablet_manager.cc:1398] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.106s user 0.005s sys 0.003s
W20240503 14:05:06.878902 6435 leader_election.cc:341] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.879479 6435 leader_election.cc:302] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.880609 6551 raft_consensus.cc:2726] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:06.877933 6513 ts_tablet_manager.cc:1367] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.224s user 0.016s sys 0.000s
I20240503 14:05:06.877821 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:06.881098 6512 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
W20240503 14:05:06.882390 6289 leader_election.cc:341] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:05:06.889525 6286 leader_election.cc:341] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:06.898986 6514 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:06.900693 6514 ts_tablet_manager.cc:1367] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.254s user 0.018s sys 0.010s
I20240503 14:05:06.901686 6362 leader_election.cc:302] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:05:06.914074 6220 catalog_manager.cc:5484] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:06.913906 6514 raft_consensus.cc:348] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.915575 6514 raft_consensus.cc:374] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.915917 6514 raft_consensus.cc:729] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.916879 6514 consensus_queue.cc:260] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.918493 6514 ts_tablet_manager.cc:1398] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240503 14:05:06.919368 6514 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:06.898334 6513 raft_consensus.cc:348] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.927256 6513 raft_consensus.cc:374] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:06.927645 6513 raft_consensus.cc:729] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:06.926219 6914 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20240503 14:05:06.931360 6513 consensus_queue.cc:260] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.934798 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.935492 6397 raft_consensus.cc:2454] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:06.935807 6513 ts_tablet_manager.cc:1398] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20240503 14:05:06.936811 6513 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:06.938117 6914 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:06.940071 6915 raft_consensus.cc:2781] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:06.940774 6915 raft_consensus.cc:686] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:06.941690 6915 consensus_queue.cc:237] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.943770 6286 leader_election.cc:302] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:05:06.946995 6671 raft_consensus.cc:2781] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:06.947506 6671 raft_consensus.cc:482] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:06.947903 6671 raft_consensus.cc:3037] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.966758 6512 tablet_bootstrap.cc:654] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:06.980216 6551 raft_consensus.cc:482] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.980823 6551 raft_consensus.cc:504] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.982738 6551 leader_election.cc:288] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:06.986280 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:06.988453 6914 raft_consensus.cc:482] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:06.990921 6914 raft_consensus.cc:504] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.990962 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19868aca07442c4bc78952934029bb9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:06.992660 6321 raft_consensus.cc:2379] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in current term 1: Already voted for candidate 90a66da3b59e4511bdc1d35930fe558d in this term.
I20240503 14:05:06.996860 6436 leader_election.cc:302] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:06.997684 6901 raft_consensus.cc:3037] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:06.997570 6671 raft_consensus.cc:504] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:06.993175 6914 leader_election.cc:288] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:06.994154 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.005753 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:07.006489 6470 raft_consensus.cc:2454] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 0.
W20240503 14:05:07.007462 6362 leader_election.cc:341] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.009647 6364 leader_election.cc:302] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:07.006510 6514 tablet_bootstrap.cc:654] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.011874 6913 raft_consensus.cc:2781] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:07.012355 6913 raft_consensus.cc:482] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:07.012725 6913 raft_consensus.cc:3037] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.020488 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:07.021605 6671 leader_election.cc:288] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.021764 6394 raft_consensus.cc:3037] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.023166 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:07.026271 6901 raft_consensus.cc:2726] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:05:07.029626 6289 leader_election.cc:341] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.052855 6394 raft_consensus.cc:2454] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:07.054832 6286 leader_election.cc:302] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1
I20240503 14:05:07.054880 6513 tablet_bootstrap.cc:654] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.059700 6671 raft_consensus.cc:2781] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:07.060259 6671 raft_consensus.cc:686] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:07.065212 6671 consensus_queue.cc:237] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.071907 6218 catalog_manager.cc:5484] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:07.076866 6514 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:07.077489 6514 ts_tablet_manager.cc:1367] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.158s user 0.020s sys 0.004s
I20240503 14:05:07.079492 6913 raft_consensus.cc:504] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.084421 6512 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.084780 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
W20240503 14:05:07.086745 6362 leader_election.cc:341] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.080422 6514 raft_consensus.cc:348] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.087581 6514 raft_consensus.cc:374] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.088817 6514 raft_consensus.cc:729] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.089404 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:07.088405 6913 leader_election.cc:288] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.090042 6469 raft_consensus.cc:3037] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.089562 6514 consensus_queue.cc:260] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.084937 6512 ts_tablet_manager.cc:1367] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.206s user 0.027s sys 0.012s
I20240503 14:05:07.092365 6514 ts_tablet_manager.cc:1398] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240503 14:05:07.093451 6514 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:07.093868 6512 raft_consensus.cc:348] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.096014 6512 raft_consensus.cc:374] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.096330 6512 raft_consensus.cc:729] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.096957 6512 consensus_queue.cc:260] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.091358 6220 catalog_manager.cc:5484] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:07.100257 6512 ts_tablet_manager.cc:1398] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:05:07.101333 6512 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:07.103556 6514 tablet_bootstrap.cc:654] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.109419 6469 raft_consensus.cc:2454] T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
I20240503 14:05:07.127646 6364 leader_election.cc:302] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:07.136480 6913 raft_consensus.cc:2781] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:07.137068 6913 raft_consensus.cc:686] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:07.137897 6913 consensus_queue.cc:237] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.119217 6512 tablet_bootstrap.cc:654] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.110317 6514 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:07.174664 6512 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.191093 6512 ts_tablet_manager.cc:1367] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.000s
I20240503 14:05:07.194205 6512 raft_consensus.cc:348] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.194955 6512 raft_consensus.cc:374] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.195232 6512 raft_consensus.cc:729] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.166278 6514 ts_tablet_manager.cc:1367] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.004s
I20240503 14:05:07.195823 6512 consensus_queue.cc:260] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.228747 6512 ts_tablet_manager.cc:1398] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20240503 14:05:07.229997 6512 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:07.223031 6220 catalog_manager.cc:5484] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:07.180524 6513 tablet_bootstrap.cc:492] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:07.245452 6513 ts_tablet_manager.cc:1367] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.309s user 0.018s sys 0.001s
I20240503 14:05:07.244436 6514 raft_consensus.cc:348] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.263002 6514 raft_consensus.cc:374] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.263381 6514 raft_consensus.cc:729] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.264434 6514 consensus_queue.cc:260] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.265298 6513 raft_consensus.cc:348] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.266062 6513 raft_consensus.cc:374] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.266364 6513 raft_consensus.cc:729] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.267017 6513 consensus_queue.cc:260] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.273001 6514 ts_tablet_manager.cc:1398] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20240503 14:05:07.274293 6514 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:07.274860 6513 ts_tablet_manager.cc:1398] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240503 14:05:07.277197 6513 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:07.283169 6513 tablet_bootstrap.cc:654] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.255291 6512 tablet_bootstrap.cc:654] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.290297 6913 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.291240 6913 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.294526 6913 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.295215 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:07.296123 6470 raft_consensus.cc:2379] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:07.297843 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.298727 6316 raft_consensus.cc:2379] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 9a9ba562b78c4746b83a40ca7c30c00d in this term.
I20240503 14:05:07.299786 6362 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:07.301367 6895 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.302179 6513 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:07.302697 6513 ts_tablet_manager.cc:1367] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.001s
I20240503 14:05:07.306171 6514 tablet_bootstrap.cc:654] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.305748 6397 raft_consensus.cc:3037] T 1407f54e751147b3b05f2b4083dbc041 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.324014 6513 raft_consensus.cc:348] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.326324 6513 raft_consensus.cc:374] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.326743 6513 raft_consensus.cc:729] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.327442 6513 consensus_queue.cc:260] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.325603 6759 raft_consensus.cc:482] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.337630 6513 ts_tablet_manager.cc:1398] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.034s user 0.001s sys 0.005s
I20240503 14:05:07.340356 6895 raft_consensus.cc:2726] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:07.335377 6759 raft_consensus.cc:504] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.350666 6513 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:07.362222 6512 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.362850 6512 ts_tablet_manager.cc:1367] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.133s user 0.019s sys 0.016s
I20240503 14:05:07.366127 6512 raft_consensus.cc:348] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.366925 6512 raft_consensus.cc:374] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.367251 6512 raft_consensus.cc:729] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.368330 6512 consensus_queue.cc:260] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.368938 6759 leader_election.cc:288] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:07.370014 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665b49a99fb74a1ab8ad655e76ca0e34" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:07.371784 6512 ts_tablet_manager.cc:1398] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20240503 14:05:07.374029 6395 raft_consensus.cc:2454] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:07.375445 6435 leader_election.cc:302] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:07.376729 6901 raft_consensus.cc:2781] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:07.377338 6901 raft_consensus.cc:482] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:07.377491 6512 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:07.377882 6901 raft_consensus.cc:3037] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.376788 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665b49a99fb74a1ab8ad655e76ca0e34" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.391566 6898 consensus_queue.cc:1035] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:07.392494 6316 raft_consensus.cc:2454] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:07.393585 6901 raft_consensus.cc:504] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.397015 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665b49a99fb74a1ab8ad655e76ca0e34" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:07.397851 6398 raft_consensus.cc:3037] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.398061 6316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "665b49a99fb74a1ab8ad655e76ca0e34" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:07.398725 6316 raft_consensus.cc:3037] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.401594 6514 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:07.404273 6514 ts_tablet_manager.cc:1367] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.130s user 0.030s sys 0.005s
I20240503 14:05:07.407284 6513 tablet_bootstrap.cc:654] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.409556 6398 raft_consensus.cc:2454] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:07.395844 6901 leader_election.cc:288] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:07.411128 6435 leader_election.cc:302] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:07.412282 6901 raft_consensus.cc:2781] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:07.413328 6514 raft_consensus.cc:348] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.415131 6514 raft_consensus.cc:374] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.415863 6514 raft_consensus.cc:729] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.417603 6316 raft_consensus.cc:2454] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:07.418383 6512 tablet_bootstrap.cc:654] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.417390 6514 consensus_queue.cc:260] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.421438 6901 raft_consensus.cc:686] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:07.426903 6901 consensus_queue.cc:237] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.433346 6898 consensus_queue.cc:1035] T 1407f54e751147b3b05f2b4083dbc041 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:07.435108 6512 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.436498 6512 ts_tablet_manager.cc:1367] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.001s
I20240503 14:05:07.436829 6915 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:07.439121 6512 raft_consensus.cc:348] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.439942 6512 raft_consensus.cc:374] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.440294 6512 raft_consensus.cc:729] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.440935 6512 consensus_queue.cc:260] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.441538 6514 ts_tablet_manager.cc:1398] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20240503 14:05:07.443140 6512 ts_tablet_manager.cc:1398] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 14:05:07.444866 6512 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:07.449371 6514 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
I20240503 14:05:07.451439 6551 raft_consensus.cc:482] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.451978 6551 raft_consensus.cc:504] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.454938 6551 leader_election.cc:288] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:07.455801 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:07.457145 6512 tablet_bootstrap.cc:654] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:07.457252 6435 leader_election.cc:341] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.462661 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.463480 6321 raft_consensus.cc:2454] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:07.467451 6671 raft_consensus.cc:482] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.468029 6671 raft_consensus.cc:504] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.469342 6436 leader_election.cc:302] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:07.471374 6852 raft_consensus.cc:2781] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:07.471781 6852 raft_consensus.cc:482] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:07.475469 6852 raft_consensus.cc:3037] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.472782 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:07.485416 6472 raft_consensus.cc:2454] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 0.
I20240503 14:05:07.473615 6512 tablet_bootstrap.cc:492] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.486680 6512 ts_tablet_manager.cc:1367] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.007s
I20240503 14:05:07.488710 6289 leader_election.cc:302] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:07.489486 6897 raft_consensus.cc:2781] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:07.489908 6897 raft_consensus.cc:482] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:07.490252 6897 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.489447 6512 raft_consensus.cc:348] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.491077 6512 raft_consensus.cc:374] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.491370 6512 raft_consensus.cc:729] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.491992 6512 consensus_queue.cc:260] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.494231 6512 ts_tablet_manager.cc:1398] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:05:07.495223 6512 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d: Bootstrap starting.
I20240503 14:05:07.498332 6514 tablet_bootstrap.cc:654] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.471531 6671 leader_election.cc:288] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.505438 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:07.511713 6286 leader_election.cc:341] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.513092 6852 raft_consensus.cc:504] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.516340 6897 raft_consensus.cc:504] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.517652 6852 leader_election.cc:288] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:07.520539 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:05:07.521911 6435 leader_election.cc:341] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.523552 6513 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:07.524102 6513 ts_tablet_manager.cc:1367] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.174s user 0.027s sys 0.015s
I20240503 14:05:07.524420 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:07.525058 6321 raft_consensus.cc:3037] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.526008 6513 raft_consensus.cc:348] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.526506 6513 raft_consensus.cc:374] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.526666 6513 raft_consensus.cc:729] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.527081 6513 consensus_queue.cc:260] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.529276 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:07.529882 6470 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.534987 6395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:07.535671 6395 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:07.538906 6513 ts_tablet_manager.cc:1398] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240503 14:05:07.540292 6513 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
I20240503 14:05:07.541494 6512 tablet_bootstrap.cc:654] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.547739 6218 catalog_manager.cc:5484] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:07.556888 6895 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:07.522207 6897 leader_election.cc:288] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.587121 6513 tablet_bootstrap.cc:654] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:07.597128 6470 raft_consensus.cc:2454] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:07.606679 6395 raft_consensus.cc:2454] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:07.596824 6321 raft_consensus.cc:2454] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:07.607753 6915 raft_consensus.cc:482] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.580397 6897 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240503 14:05:07.617734 6286 leader_election.cc:302] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:07.620311 6671 raft_consensus.cc:2781] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:07.620904 6671 raft_consensus.cc:686] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:07.621711 6671 consensus_queue.cc:237] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:07.628867 6435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 80205.5us
I20240503 14:05:07.610268 6915 raft_consensus.cc:504] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:07.653681 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 68399us
I20240503 14:05:07.657060 6436 leader_election.cc:302] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:07.675168 6512 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d: No bootstrap required, opened a new log
I20240503 14:05:07.675834 6512 ts_tablet_manager.cc:1367] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent bootstrapping tablet: real 0.181s user 0.014s sys 0.013s
I20240503 14:05:07.677424 6759 raft_consensus.cc:2781] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:07.691089 6759 raft_consensus.cc:686] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:07.692034 6759 consensus_queue.cc:237] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.692469 6895 consensus_queue.cc:1035] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:07.697851 6514 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:07.698455 6514 ts_tablet_manager.cc:1367] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.254s user 0.013s sys 0.032s
W20240503 14:05:07.705981 6362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 86040.8us
I20240503 14:05:07.713824 6514 raft_consensus.cc:348] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.714771 6514 raft_consensus.cc:374] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.715047 6514 raft_consensus.cc:729] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.715744 6514 consensus_queue.cc:260] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:07.723507 6936 log.cc:924] Time spent T 1407f54e751147b3b05f2b4083dbc041 P 8e7d4e1e07e94f21b75ba42bf86334e1: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20240503 14:05:07.724119 6514 ts_tablet_manager.cc:1398] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20240503 14:05:07.726769 6772 raft_consensus.cc:482] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.727424 6772 raft_consensus.cc:504] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.728338 6915 leader_election.cc:288] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.729735 6772 leader_election.cc:288] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:07.733413 6512 raft_consensus.cc:348] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.734491 6512 raft_consensus.cc:374] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.734797 6512 raft_consensus.cc:729] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.745651 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:07.745591 6512 consensus_queue.cc:260] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.755153 6514 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Bootstrap starting.
W20240503 14:05:07.755772 6435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 120800us
I20240503 14:05:07.763340 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.774868 6513 tablet_bootstrap.cc:492] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:07.776485 6513 ts_tablet_manager.cc:1367] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.236s user 0.011s sys 0.014s
I20240503 14:05:07.748874 6898 raft_consensus.cc:482] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.779772 6220 catalog_manager.cc:5484] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:07.785465 6551 raft_consensus.cc:482] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.785984 6551 raft_consensus.cc:504] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.794763 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:07.795550 6323 raft_consensus.cc:2454] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:07.808642 6436 leader_election.cc:302] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
W20240503 14:05:07.818487 6286 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.194:39097, user_credentials={real_user=slave}} blocked reactor thread for 69319.2us
W20240503 14:05:07.834793 6286 leader_election.cc:341] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:07.847147 6551 leader_election.cc:288] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:07.850867 6512 ts_tablet_manager.cc:1398] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d: Time spent starting tablet: real 0.169s user 0.012s sys 0.007s
I20240503 14:05:07.820886 6759 raft_consensus.cc:2781] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:07.858167 6513 raft_consensus.cc:348] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.859030 6513 raft_consensus.cc:374] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:07.859330 6513 raft_consensus.cc:729] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:07.790531 6472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:07.870437 6513 consensus_queue.cc:260] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.790259 6898 raft_consensus.cc:504] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:07.863382 6759 raft_consensus.cc:482] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:05:07.889441 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 80915.6us
I20240503 14:05:07.885339 6759 raft_consensus.cc:3037] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:05:07.927268 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 265456us
I20240503 14:05:07.895236 6513 ts_tablet_manager.cc:1398] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.118s user 0.006s sys 0.001s
I20240503 14:05:07.975116 6221 catalog_manager.cc:5484] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:07.941756 6898 leader_election.cc:288] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.013797 6398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:07.894726 6316 raft_consensus.cc:3037] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.023039 6513 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
W20240503 14:05:08.026525 6289 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 262298us
I20240503 14:05:07.989508 6958 raft_consensus.cc:482] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:07.920269 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:08.036290 6470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:08.037040 6470 raft_consensus.cc:2454] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 1.
W20240503 14:05:08.050550 6269 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240503 14:05:08.040410 6396 raft_consensus.cc:2454] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:08.036180 6958 raft_consensus.cc:504] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.058714 6514 tablet_bootstrap.cc:654] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Neither blocks nor log segments found. Creating new log.
W20240503 14:05:08.057497 6289 consensus_peers.cc:485] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d -> Peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Couldn't send request to peer 8e7d4e1e07e94f21b75ba42bf86334e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:08.054271 6942 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20240503 14:05:08.052376 6759 raft_consensus.cc:504] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.080560 6941 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:08.081733 6472 raft_consensus.cc:2379] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a9ba562b78c4746b83a40ca7c30c00d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:08.095909 6513 tablet_bootstrap.cc:654] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:08.100001 6958 leader_election.cc:288] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:08.081796 6759 leader_election.cc:288] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:08.119814 6397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:08.113451 6955 raft_consensus.cc:482] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:08.121881 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:08.136946 6364 leader_election.cc:302] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:08.141407 6321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8cbd74268fe4e5d8dc5e5d3cb1ed6d1" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:08.142493 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc21dfebd854cb3b4f5ce3fe803b0fc" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:08.143208 6318 raft_consensus.cc:3037] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.143395 6394 raft_consensus.cc:3037] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.147182 6913 raft_consensus.cc:2781] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:05:08.147675 6913 raft_consensus.cc:482] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:08.148036 6913 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.127439 6955 raft_consensus.cc:504] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
W20240503 14:05:08.168915 6435 consensus_peers.cc:485] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Couldn't send request to peer 90a66da3b59e4511bdc1d35930fe558d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:08.197980 6394 raft_consensus.cc:2454] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:08.204195 6435 leader_election.cc:302] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters:
I20240503 14:05:08.210215 6913 raft_consensus.cc:504] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.218076 6955 leader_election.cc:288] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.221172 6317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:08.197892 6318 raft_consensus.cc:2454] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:08.251324 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7b3d6ea1b264b0e8781218ca4ee5155" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:08.237643 6852 raft_consensus.cc:2781] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:08.258358 6913 leader_election.cc:288] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.240518 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:08.265014 6852 raft_consensus.cc:686] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:08.265934 6852 consensus_queue.cc:237] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.278566 6514 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1: No bootstrap required, opened a new log
I20240503 14:05:08.279561 6396 raft_consensus.cc:1226] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Rejecting Update request from peer 9a9ba562b78c4746b83a40ca7c30c00d for earlier term 1. Current term is 2. Ops: []
I20240503 14:05:08.276731 6466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf848b599cbc45ecb8772ecfb50276e7" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:08.281613 6466 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.283594 6514 ts_tablet_manager.cc:1367] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent bootstrapping tablet: real 0.529s user 0.018s sys 0.033s
I20240503 14:05:08.288748 6289 leader_election.cc:302] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:08.293011 6289 leader_election.cc:302] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:08.296814 6913 consensus_queue.cc:1035] T c7b3d6ea1b264b0e8781218ca4ee5155 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:08.298118 6471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
W20240503 14:05:08.289222 6286 leader_election.cc:341] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:08.303586 6772 consensus_queue.cc:1046] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:08.304440 6467 raft_consensus.cc:3037] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.308367 6513 tablet_bootstrap.cc:492] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:08.309082 6513 ts_tablet_manager.cc:1367] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.296s user 0.011s sys 0.022s
I20240503 14:05:08.312662 6970 raft_consensus.cc:2726] T c7b3d6ea1b264b0e8781218ca4ee5155 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:08.313668 6514 raft_consensus.cc:348] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.314467 6514 raft_consensus.cc:374] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:08.314757 6514 raft_consensus.cc:729] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Initialized, Role: FOLLOWER
I20240503 14:05:08.318944 6513 raft_consensus.cc:348] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.319934 6513 raft_consensus.cc:374] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:08.321744 6513 raft_consensus.cc:729] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:08.315624 6514 consensus_queue.cc:260] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.308105 6955 raft_consensus.cc:2726] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:08.322295 6513 consensus_queue.cc:260] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.332911 6436 leader_election.cc:302] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1; no voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:08.346731 6513 ts_tablet_manager.cc:1398] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20240503 14:05:08.356068 6514 ts_tablet_manager.cc:1398] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Time spent starting tablet: real 0.072s user 0.007s sys 0.000s
I20240503 14:05:08.366382 6513 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d: Bootstrap starting.
W20240503 14:05:08.367233 6289 leader_election.cc:341] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:08.371258 6289 leader_election.cc:302] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:08.373485 6898 raft_consensus.cc:2726] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:08.376300 6671 raft_consensus.cc:3032] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:05:08.376731 6671 raft_consensus.cc:729] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:08.377480 6671 consensus_queue.cc:260] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.378574 6671 raft_consensus.cc:3037] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.386294 6218 catalog_manager.cc:5484] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
W20240503 14:05:08.400514 6974 log.cc:924] Time spent T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20240503 14:05:08.410657 6939 raft_consensus.cc:482] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:08.411226 6939 raft_consensus.cc:504] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.412940 6939 leader_election.cc:288] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:08.414126 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
W20240503 14:05:08.415652 6435 leader_election.cc:341] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:08.428330 6466 raft_consensus.cc:2454] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90a66da3b59e4511bdc1d35930fe558d in term 2.
I20240503 14:05:08.429112 6468 raft_consensus.cc:1226] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Rejecting Update request from peer 9a9ba562b78c4746b83a40ca7c30c00d for earlier term 1. Current term is 2. Ops: []
I20240503 14:05:08.431911 6364 leader_election.cc:302] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 90a66da3b59e4511bdc1d35930fe558d; no voters: 9a9ba562b78c4746b83a40ca7c30c00d
W20240503 14:05:08.433998 6972 log.cc:924] Time spent T c7b3d6ea1b264b0e8781218ca4ee5155 P 8e7d4e1e07e94f21b75ba42bf86334e1: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20240503 14:05:08.436708 6946 raft_consensus.cc:2781] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:05:08.438701 6946 raft_consensus.cc:686] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 LEADER]: Becoming Leader. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Running, Role: LEADER
I20240503 14:05:08.439656 6946 consensus_queue.cc:237] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.466275 6513 tablet_bootstrap.cc:654] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d: Neither blocks nor log segments found. Creating new log.
I20240503 14:05:08.475673 6218 catalog_manager.cc:5484] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: term changed from 1 to 2, leader changed from 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) to 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194). New cstate: current_term: 2 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:08.477185 6551 raft_consensus.cc:2726] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:05:08.474844 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 104184us
I20240503 14:05:08.485777 6319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:08.486563 6319 raft_consensus.cc:2454] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 0.
I20240503 14:05:08.493508 6436 leader_election.cc:302] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:08.494426 6969 consensus_queue.cc:1035] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 14:05:08.503542 6551 raft_consensus.cc:2781] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:05:08.503964 6551 raft_consensus.cc:482] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:08.504305 6551 raft_consensus.cc:3037] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.520190 6551 raft_consensus.cc:504] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.521993 6551 leader_election.cc:288] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Requested vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:08.523066 6394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
W20240503 14:05:08.524642 6435 leader_election.cc:341] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:05:08.524745 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d"
I20240503 14:05:08.525354 6322 raft_consensus.cc:3037] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.530478 6322 raft_consensus.cc:2454] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in term 1.
I20240503 14:05:08.531958 6436 leader_election.cc:302] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d; no voters: 90a66da3b59e4511bdc1d35930fe558d
I20240503 14:05:08.533814 6852 raft_consensus.cc:2781] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:05:08.534286 6852 raft_consensus.cc:686] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Becoming Leader. State: Replica: 8e7d4e1e07e94f21b75ba42bf86334e1, State: Running, Role: LEADER
I20240503 14:05:08.535158 6852 consensus_queue.cc:237] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.547339 6946 raft_consensus.cc:482] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:08.549412 6946 raft_consensus.cc:504] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.553714 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:08.556453 6946 leader_election.cc:288] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.552537 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ce484e160849488b2637359d07ba88" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:08.560530 6362 leader_election.cc:302] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:08.561332 6946 raft_consensus.cc:2726] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:08.568169 6513 tablet_bootstrap.cc:492] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d: No bootstrap required, opened a new log
I20240503 14:05:08.568666 6513 ts_tablet_manager.cc:1367] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent bootstrapping tablet: real 0.215s user 0.012s sys 0.002s
I20240503 14:05:08.571166 6513 raft_consensus.cc:348] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.575121 6218 catalog_manager.cc:5484] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195). New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240503 14:05:08.571822 6513 raft_consensus.cc:374] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:05:08.584020 6513 raft_consensus.cc:729] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a66da3b59e4511bdc1d35930fe558d, State: Initialized, Role: FOLLOWER
I20240503 14:05:08.584846 6513 consensus_queue.cc:260] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.597354 6513 ts_tablet_manager.cc:1398] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240503 14:05:08.764279 6396 raft_consensus.cc:3037] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.791642 6551 consensus_queue.cc:1035] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:08.812381 6759 consensus_queue.cc:1035] T 0bc21dfebd854cb3b4f5ce3fe803b0fc P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:08.881140 6946 raft_consensus.cc:482] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:08.881659 6946 raft_consensus.cc:504] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.885342 6946 leader_election.cc:288] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.885476 6322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" is_pre_election: true
I20240503 14:05:08.886225 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75042082bf74978bd28f824f5bee4e9" candidate_uuid: "90a66da3b59e4511bdc1d35930fe558d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:08.887007 6322 raft_consensus.cc:2379] T e75042082bf74978bd28f824f5bee4e9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90a66da3b59e4511bdc1d35930fe558d in current term 1: Already voted for candidate 8e7d4e1e07e94f21b75ba42bf86334e1 in this term.
I20240503 14:05:08.888823 6362 leader_election.cc:302] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d; no voters: 8e7d4e1e07e94f21b75ba42bf86334e1, 9a9ba562b78c4746b83a40ca7c30c00d
I20240503 14:05:08.888978 6964 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:05:08.889976 6946 raft_consensus.cc:3037] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:05:08.889684 6964 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.892334 6964 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.892592 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d" is_pre_election: true
I20240503 14:05:08.893779 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" is_pre_election: true
I20240503 14:05:08.894613 6396 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:08.894737 6467 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 1.
I20240503 14:05:08.895844 6286 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:08.896519 6946 raft_consensus.cc:2726] T e75042082bf74978bd28f824f5bee4e9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:05:08.896735 6671 raft_consensus.cc:2781] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:05:08.897152 6671 raft_consensus.cc:482] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:05:08.897439 6671 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.904001 6671 raft_consensus.cc:504] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.906690 6396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a66da3b59e4511bdc1d35930fe558d"
I20240503 14:05:08.906862 6671 leader_election.cc:288] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 election: Requested vote from peers 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097), 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:08.907411 6396 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.911495 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9477869b1b604feb8f56531957fb13e6" candidate_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1"
I20240503 14:05:08.912237 6467 raft_consensus.cc:3037] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:05:08.914019 6396 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 2.
I20240503 14:05:08.915061 6286 leader_election.cc:302] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a66da3b59e4511bdc1d35930fe558d, 9a9ba562b78c4746b83a40ca7c30c00d; no voters:
I20240503 14:05:08.917388 6322 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240503 14:05:08.917887 6322 pending_rounds.cc:76] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Aborting all ops after (but not including) 0
I20240503 14:05:08.918215 6322 pending_rounds.cc:97] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240503 14:05:08.918617 6322 raft_consensus.cc:2866] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:05:08.918815 6671 raft_consensus.cc:2781] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:05:08.919512 6671 raft_consensus.cc:686] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 LEADER]: Becoming Leader. State: Replica: 9a9ba562b78c4746b83a40ca7c30c00d, State: Running, Role: LEADER
I20240503 14:05:08.920297 6671 consensus_queue.cc:237] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } }
I20240503 14:05:08.920506 6946 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:08.923036 6467 raft_consensus.cc:2454] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9ba562b78c4746b83a40ca7c30c00d in term 2.
I20240503 14:05:08.944378 6467 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 14:05:08.945896 6946 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:08.946260 6220 catalog_manager.cc:5484] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: term changed from 0 to 2, leader changed from <none> to 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193). New cstate: current_term: 2 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:08.979460 6960 consensus_queue.cc:1035] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:08.994621 6551 consensus_queue.cc:1035] T e75042082bf74978bd28f824f5bee4e9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:09.198897 5099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:05:09.231463 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:09.232786 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:05:09.238883 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:05:09.243288 5099 server_base.cc:1034] running on GCE node
I20240503 14:05:09.245260 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:05:09.245617 5099 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:05:09.245852 5099 hybrid_clock.cc:648] HybridClock initialized: now 1714745109245832 us; error 0 us; skew 500 ppm
I20240503 14:05:09.246665 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:05:09.262065 5099 webserver.cc:460] Webserver started at http://127.4.250.196:42943/ using document root <none> and password file <none>
I20240503 14:05:09.262971 5099 fs_manager.cc:359] Metadata directory not provided
I20240503 14:05:09.263365 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:05:09.263893 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:05:09.265813 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-3-root/instance:
uuid: "64a069ee90f74166b112f45b458364dc"
format_stamp: "Formatted at 2024-05-03 14:05:09 on dist-test-slave-8wrt"
I20240503 14:05:09.272925 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240503 14:05:09.278499 7005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:09.279806 5099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:05:09.280238 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-3-root
uuid: "64a069ee90f74166b112f45b458364dc"
format_stamp: "Formatted at 2024-05-03 14:05:09 on dist-test-slave-8wrt"
I20240503 14:05:09.280702 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:05:09.298655 5099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:05:09.300168 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:05:09.301967 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:05:09.313414 5099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:05:09.313867 5099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 14:05:09.314342 5099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:05:09.314666 5099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:05:09.334728 6467 raft_consensus.cc:1261] T 9477869b1b604feb8f56531957fb13e6 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 14:05:09.340287 6964 consensus_queue.cc:1035] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:05:09.385051 6396 raft_consensus.cc:1261] T 9477869b1b604feb8f56531957fb13e6 P 90a66da3b59e4511bdc1d35930fe558d [term 2 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 14:05:09.386828 6955 consensus_queue.cc:1035] T 9477869b1b604feb8f56531957fb13e6 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:05:09.795606 5099 rpc_server.cc:307] RPC server started. Bound to: 127.4.250.196:46183
I20240503 14:05:09.798923 7070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.250.196:46183 every 8 connection(s)
I20240503 14:05:09.808642 5099 auto_leader_rebalancer-test.cc:405] add a tserver: 64a069ee90f74166b112f45b458364dc
I20240503 14:05:09.831610 7071 heartbeater.cc:347] Connected to a master server at 127.4.250.254:36245
I20240503 14:05:09.832334 7071 heartbeater.cc:464] Registering TS with master...
I20240503 14:05:09.833479 7071 heartbeater.cc:511] Master 127.4.250.254:36245 requested a full tablet report, sending...
I20240503 14:05:09.835793 6218 ts_manager.cc:194] Registered new tserver with Master: 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183)
I20240503 14:05:09.837389 6218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44994
I20240503 14:05:10.223542 6397 consensus_queue.cc:237] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:10.245994 6317 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.248116 7051 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:10.259157 6467 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.261081 7051 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:10.270393 7051 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.334597 6317 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.346423 6466 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.346578 6218 catalog_manager.cc:5484] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:10.365541 6469 consensus_queue.cc:237] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
W20240503 14:05:10.391223 6364 consensus_peers.cc:485] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0285fa210834b8c9b116d0cf7bb2430. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:10.415130 6397 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.419682 6321 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.419883 7092 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:10.424911 6960 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:10.452302 6969 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.460902 6397 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
W20240503 14:05:10.510155 6438 consensus_peers.cc:485] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 665b49a99fb74a1ab8ad655e76ca0e34. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:10.521651 6317 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.554049 6321 consensus_queue.cc:237] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:10.550958 6220 catalog_manager.cc:5484] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.583662 6466 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.587175 6671 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:10.592612 6394 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.599843 6671 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 14:05:10.629990 6289 consensus_peers.cc:485] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8cbd74268fe4e5d8dc5e5d3cb1ed6d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:10.637315 6671 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.643043 6466 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.661932 7103 ts_tablet_manager.cc:902] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:10.677175 6397 consensus_queue.cc:237] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:10.679360 6220 catalog_manager.cc:5484] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:10.694540 7103 tablet_copy_client.cc:308] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
W20240503 14:05:10.700134 6364 consensus_peers.cc:485] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a19868aca07442c4bc78952934029bb9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:10.700986 6466 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.703733 7081 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:10.706360 6397 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.721555 6321 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:10.723275 7086 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:10.742588 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet b0285fa210834b8c9b116d0cf7bb2430 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:10.743691 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet b0285fa210834b8c9b116d0cf7bb2430 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-b0285fa210834b8c9b116d0cf7bb2430
I20240503 14:05:10.747609 7085 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.756338 6466 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.755079 6317 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:10.802749 6220 catalog_manager.cc:5484] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:10.819569 6408 tablet_copy_source_session.cc:205] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:10.827383 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0285fa210834b8c9b116d0cf7bb2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:10.840754 7071 heartbeater.cc:503] Master 127.4.250.254:36245 was elected leader, sending a full tablet report...
I20240503 14:05:10.863879 7103 tablet_copy_client.cc:783] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:10.865011 7103 tablet_copy_client.cc:647] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:10.874600 7103 tablet_copy_client.cc:523] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:10.895385 7103 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:10.950415 7103 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:10.952405 7103 tablet_bootstrap.cc:492] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:10.956766 7115 ts_tablet_manager.cc:902] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:10.953553 7103 ts_tablet_manager.cc:1367] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.058s user 0.031s sys 0.021s
I20240503 14:05:10.963541 7103 raft_consensus.cc:348] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:10.965070 7103 raft_consensus.cc:729] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:10.966109 7103 consensus_queue.cc:260] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.015473 7103 ts_tablet_manager.cc:1398] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.055s user 0.005s sys 0.021s
I20240503 14:05:11.023878 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-b0285fa210834b8c9b116d0cf7bb2430 received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:11.024554 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-b0285fa210834b8c9b116d0cf7bb2430 on tablet b0285fa210834b8c9b116d0cf7bb2430 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:11.031299 7103 ts_tablet_manager.cc:902] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155)
I20240503 14:05:11.099651 7115 tablet_copy_client.cc:308] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.195:35583
I20240503 14:05:11.138252 7103 tablet_copy_client.cc:308] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.193:37155
I20240503 14:05:11.138715 6482 tablet_copy_service.cc:140] P 8e7d4e1e07e94f21b75ba42bf86334e1: Received BeginTabletCopySession request for tablet 665b49a99fb74a1ab8ad655e76ca0e34 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:40718)
I20240503 14:05:11.139377 6482 tablet_copy_service.cc:161] P 8e7d4e1e07e94f21b75ba42bf86334e1: Beginning new tablet copy session on tablet 665b49a99fb74a1ab8ad655e76ca0e34 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:40718: session id = 64a069ee90f74166b112f45b458364dc-665b49a99fb74a1ab8ad655e76ca0e34
I20240503 14:05:11.149459 6482 tablet_copy_source_session.cc:205] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:11.154400 7115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b49a99fb74a1ab8ad655e76ca0e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.158883 6333 tablet_copy_service.cc:140] P 9a9ba562b78c4746b83a40ca7c30c00d: Received BeginTabletCopySession request for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:59810)
I20240503 14:05:11.159680 6333 tablet_copy_service.cc:161] P 9a9ba562b78c4746b83a40ca7c30c00d: Beginning new tablet copy session on tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:59810: session id = 64a069ee90f74166b112f45b458364dc-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1
I20240503 14:05:11.175832 6333 tablet_copy_source_session.cc:205] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:11.181638 7115 tablet_copy_client.cc:783] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:11.182660 7115 tablet_copy_client.cc:647] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:11.191246 7115 tablet_copy_client.cc:523] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:11.196434 7103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.204016 7115 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:11.218712 7103 tablet_copy_client.cc:783] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:11.219470 7103 tablet_copy_client.cc:647] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:11.236383 7044 raft_consensus.cc:1203] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:11.241081 7103 tablet_copy_client.cc:523] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:11.254628 7103 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:11.264503 7115 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:11.276255 7115 tablet_bootstrap.cc:492] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:11.285663 7115 ts_tablet_manager.cc:1367] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.082s user 0.023s sys 0.003s
I20240503 14:05:11.285861 7118 ts_tablet_manager.cc:902] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:11.297912 7115 raft_consensus.cc:348] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.300391 7115 raft_consensus.cc:729] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:11.306412 7115 consensus_queue.cc:260] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.320211 7115 ts_tablet_manager.cc:1398] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.034s user 0.009s sys 0.001s
I20240503 14:05:11.326309 6482 tablet_copy_service.cc:342] P 8e7d4e1e07e94f21b75ba42bf86334e1: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-665b49a99fb74a1ab8ad655e76ca0e34 received from {username='slave'} at 127.0.0.1:40718
I20240503 14:05:11.326779 6482 tablet_copy_service.cc:433] P 8e7d4e1e07e94f21b75ba42bf86334e1: ending tablet copy session 64a069ee90f74166b112f45b458364dc-665b49a99fb74a1ab8ad655e76ca0e34 on tablet 665b49a99fb74a1ab8ad655e76ca0e34 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:11.336769 7103 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:11.337801 7103 tablet_bootstrap.cc:492] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:11.338572 7103 ts_tablet_manager.cc:1367] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.012s
I20240503 14:05:11.340523 7118 tablet_copy_client.cc:308] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
I20240503 14:05:11.342530 7103 raft_consensus.cc:348] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.343215 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet a19868aca07442c4bc78952934029bb9 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:11.343510 7103 raft_consensus.cc:729] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:11.343777 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet a19868aca07442c4bc78952934029bb9 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-a19868aca07442c4bc78952934029bb9
I20240503 14:05:11.344257 7103 consensus_queue.cc:260] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.352598 7103 ts_tablet_manager.cc:1398] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20240503 14:05:11.354557 6333 tablet_copy_service.cc:342] P 9a9ba562b78c4746b83a40ca7c30c00d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 received from {username='slave'} at 127.0.0.1:59810
I20240503 14:05:11.354972 6333 tablet_copy_service.cc:433] P 9a9ba562b78c4746b83a40ca7c30c00d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 on tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:11.373641 7044 raft_consensus.cc:1203] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:11.386121 6408 tablet_copy_source_session.cc:205] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:11.389529 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19868aca07442c4bc78952934029bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:11.409315 7118 tablet_copy_client.cc:783] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:11.410020 7118 tablet_copy_client.cc:647] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:11.423015 7118 tablet_copy_client.cc:523] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:11.432876 7044 raft_consensus.cc:1203] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:11.442601 7118 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:11.515569 7118 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:11.516496 7118 tablet_bootstrap.cc:492] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:11.517655 7118 ts_tablet_manager.cc:1367] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.082s user 0.037s sys 0.020s
I20240503 14:05:11.520167 7118 raft_consensus.cc:348] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.520933 7118 raft_consensus.cc:729] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:11.521509 7118 consensus_queue.cc:260] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:11.523924 7118 ts_tablet_manager.cc:1398] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:05:11.527515 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-a19868aca07442c4bc78952934029bb9 received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:11.527957 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-a19868aca07442c4bc78952934029bb9 on tablet a19868aca07442c4bc78952934029bb9 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:11.755708 7051 raft_consensus.cc:1050] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:11.758705 7051 consensus_queue.cc:237] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:11.766402 7044 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:11.767565 6466 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:11.767750 7051 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:11.769125 7081 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:11.770414 6321 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:11.771729 7081 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:11.780889 7081 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.782773 7044 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.802134 7043 raft_consensus.cc:1203] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:11.830195 6218 catalog_manager.cc:5484] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.842306 6466 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.840338 6317 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.881596 6394 consensus_queue.cc:237] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:11.887751 7044 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:11.889603 7081 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:11.895061 6321 raft_consensus.cc:1261] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:11.900665 7136 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.904222 7135 consensus_queue.cc:1035] T b0285fa210834b8c9b116d0cf7bb2430 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:11.926379 7044 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.939575 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0285fa210834b8c9b116d0cf7bb2430 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:11.940454 6317 raft_consensus.cc:2932] T b0285fa210834b8c9b116d0cf7bb2430 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.944407 7137 raft_consensus.cc:1050] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:11.946938 7137 consensus_queue.cc:237] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:11.968535 6218 catalog_manager.cc:5484] T b0285fa210834b8c9b116d0cf7bb2430 P 64a069ee90f74166b112f45b458364dc reported cstate change: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:11.980944 7089 raft_consensus.cc:1050] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:11.984279 7089 consensus_queue.cc:237] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:11.993542 6466 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:11.995591 7137 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.013651 6455 tablet_service.cc:1514] Processing DeleteTablet for tablet b0285fa210834b8c9b116d0cf7bb2430 with delete_type TABLET_DATA_TOMBSTONED (TS 8e7d4e1e07e94f21b75ba42bf86334e1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58394
I20240503 14:05:12.017657 7148 tablet_replica.cc:330] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: stopping tablet replica
I20240503 14:05:12.010778 7044 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.022831 6955 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.027345 6394 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.028949 6955 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.036896 7148 raft_consensus.cc:2229] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:12.037982 7148 raft_consensus.cc:2258] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:12.040709 7137 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.045495 7148 ts_tablet_manager.cc:1875] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:12.064723 7044 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.081213 7043 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.084724 6321 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.084985 7089 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.092900 7089 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.096293 6394 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.102869 6396 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.112938 7148 ts_tablet_manager.cc:1888] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:12.115166 6969 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.113597 7148 log.cc:1196] T b0285fa210834b8c9b116d0cf7bb2430 P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/wals/b0285fa210834b8c9b116d0cf7bb2430
I20240503 14:05:12.123466 6208 catalog_manager.cc:4830] TS 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): tablet b0285fa210834b8c9b116d0cf7bb2430 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:12.139925 6218 catalog_manager.cc:5484] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.158878 6969 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.174394 6469 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.191839 6316 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.184878 6317 consensus_queue.cc:237] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:12.217160 6396 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.219954 7043 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.237722 7043 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.244239 7136 raft_consensus.cc:1050] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:12.267469 6469 raft_consensus.cc:1261] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.245548 6220 catalog_manager.cc:5484] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:12.268476 7136 consensus_queue.cc:237] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:12.284551 6955 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:12.300387 7155 consensus_queue.cc:1035] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:12.323565 6955 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.341404 6469 consensus_queue.cc:237] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:12.358938 6206 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:05:12.362305 6364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 65215.1us
I20240503 14:05:12.366859 6394 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.368558 7043 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.375114 7044 raft_consensus.cc:1261] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.376274 7156 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:12.400493 6960 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.411162 6219 catalog_manager.cc:5484] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:12.416074 6467 raft_consensus.cc:2932] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.417922 7156 consensus_queue.cc:1035] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20240503 14:05:12.427738 7043 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.430253 6317 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.432731 7081 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.433934 7086 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.434733 6386 tablet_service.cc:1514] Processing DeleteTablet for tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 with delete_type TABLET_DATA_TOMBSTONED (TS 90a66da3b59e4511bdc1d35930fe558d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52860
I20240503 14:05:12.439735 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 665b49a99fb74a1ab8ad655e76ca0e34 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:12.444854 6393 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.456740 6468 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:12.444504 7043 raft_consensus.cc:2932] T 665b49a99fb74a1ab8ad655e76ca0e34 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.466689 7135 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:12.452176 6219 catalog_manager.cc:5484] T 665b49a99fb74a1ab8ad655e76ca0e34 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:12.494769 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet 665b49a99fb74a1ab8ad655e76ca0e34 with delete_type TABLET_DATA_TOMBSTONED (TS 9a9ba562b78c4746b83a40ca7c30c00d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34752
I20240503 14:05:12.524696 7164 tablet_replica.cc:330] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: stopping tablet replica
I20240503 14:05:12.526301 7164 raft_consensus.cc:2229] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:12.532444 7164 raft_consensus.cc:2258] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:12.596163 7163 tablet_replica.cc:330] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: stopping tablet replica
I20240503 14:05:12.597767 7163 raft_consensus.cc:2229] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:12.598682 7163 raft_consensus.cc:2258] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:12.616417 7164 ts_tablet_manager.cc:1875] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:12.633428 7081 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.635048 6469 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.639005 7163 ts_tablet_manager.cc:1875] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:12.666735 6316 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.678910 7044 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.682560 6219 catalog_manager.cc:5484] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.713261 6397 consensus_queue.cc:237] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:12.718294 7163 ts_tablet_manager.cc:1888] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:12.718782 7163 log.cc:1196] T a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 P 90a66da3b59e4511bdc1d35930fe558d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/wals/a8cbd74268fe4e5d8dc5e5d3cb1ed6d1
I20240503 14:05:12.720587 6205 catalog_manager.cc:4830] TS 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): tablet a8cbd74268fe4e5d8dc5e5d3cb1ed6d1 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:12.699326 7164 ts_tablet_manager.cc:1888] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:12.722102 7164 log.cc:1196] T 665b49a99fb74a1ab8ad655e76ca0e34 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/wals/665b49a99fb74a1ab8ad655e76ca0e34
I20240503 14:05:12.733285 6316 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.733392 7044 raft_consensus.cc:1261] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:12.735147 6206 catalog_manager.cc:4830] TS 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): tablet 665b49a99fb74a1ab8ad655e76ca0e34 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:12.735764 7081 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:12.737660 7081 consensus_queue.cc:1035] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:12.744843 7051 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.755725 6322 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.762867 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a19868aca07442c4bc78952934029bb9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:12.781121 6221 catalog_manager.cc:5484] T a19868aca07442c4bc78952934029bb9 P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.791939 7044 raft_consensus.cc:2932] T a19868aca07442c4bc78952934029bb9 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:12.803822 6455 tablet_service.cc:1514] Processing DeleteTablet for tablet a19868aca07442c4bc78952934029bb9 with delete_type TABLET_DATA_TOMBSTONED (TS 8e7d4e1e07e94f21b75ba42bf86334e1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58394
I20240503 14:05:12.837769 7170 tablet_replica.cc:330] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: stopping tablet replica
I20240503 14:05:12.839120 7170 raft_consensus.cc:2229] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:12.840101 7170 raft_consensus.cc:2258] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:12.843878 7170 ts_tablet_manager.cc:1875] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:12.871620 7170 ts_tablet_manager.cc:1888] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:12.872181 7170 log.cc:1196] T a19868aca07442c4bc78952934029bb9 P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/wals/a19868aca07442c4bc78952934029bb9
I20240503 14:05:12.874063 6208 catalog_manager.cc:4830] TS 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): tablet a19868aca07442c4bc78952934029bb9 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:13.894034 6322 consensus_queue.cc:237] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
W20240503 14:05:13.894091 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 54198.5us
W20240503 14:05:13.911219 6289 consensus_peers.cc:485] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a594e34044584aa09a50d15954cbb81b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:13.912631 6396 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:13.913900 6467 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:13.913969 7161 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:13.916844 7161 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:13.930161 7105 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:13.931377 6397 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:13.948244 6467 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:13.955180 6220 catalog_manager.cc:5484] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.026528 6396 consensus_queue.cc:237] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
W20240503 14:05:14.046082 6364 consensus_peers.cc:485] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3586823a1bd546f08b8007d4b55ef43e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:14.046924 6322 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:14.048569 6467 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:14.048949 7136 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:14.050724 7051 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:14.058970 7169 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.060981 6322 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.063333 6467 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.075029 6396 consensus_queue.cc:237] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.115041 6219 catalog_manager.cc:5484] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:14.155802 6468 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:14.158632 6322 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:05:14.158540 6364 consensus_peers.cc:485] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef212857f4874110a036926374b4d0cd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:14.160796 7136 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:14.163408 7169 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:14.179808 7169 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.181607 6469 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.210600 6322 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.216799 6221 catalog_manager.cc:5484] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.223250 6468 consensus_queue.cc:237] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
W20240503 14:05:14.274999 6438 consensus_peers.cc:485] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 861ddaea971348a596840c74fb1e8a71. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:14.276928 6321 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:14.281666 7173 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:14.282132 6396 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:14.286561 7171 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:14.299235 7173 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.307329 6322 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.313781 6396 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.339550 6219 catalog_manager.cc:5484] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:14.354486 7194 ts_tablet_manager.cc:902] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155)
I20240503 14:05:14.402587 7194 tablet_copy_client.cc:308] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.193:37155
I20240503 14:05:14.405427 6333 tablet_copy_service.cc:140] P 9a9ba562b78c4746b83a40ca7c30c00d: Received BeginTabletCopySession request for tablet a594e34044584aa09a50d15954cbb81b from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:59810)
I20240503 14:05:14.405973 6333 tablet_copy_service.cc:161] P 9a9ba562b78c4746b83a40ca7c30c00d: Beginning new tablet copy session on tablet a594e34044584aa09a50d15954cbb81b from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:59810: session id = 64a069ee90f74166b112f45b458364dc-a594e34044584aa09a50d15954cbb81b
I20240503 14:05:14.418880 6333 tablet_copy_source_session.cc:205] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:14.423012 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a594e34044584aa09a50d15954cbb81b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:14.450698 7194 tablet_copy_client.cc:783] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:14.451352 7194 tablet_copy_client.cc:647] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:14.464593 7194 tablet_copy_client.cc:523] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:14.474659 7194 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:14.495014 7199 ts_tablet_manager.cc:902] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:14.519233 7199 tablet_copy_client.cc:308] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
I20240503 14:05:14.520983 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet 3586823a1bd546f08b8007d4b55ef43e from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:14.521575 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet 3586823a1bd546f08b8007d4b55ef43e from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-3586823a1bd546f08b8007d4b55ef43e
I20240503 14:05:14.528857 7194 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:14.529922 7194 tablet_bootstrap.cc:492] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:14.536012 7194 ts_tablet_manager.cc:1367] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.062s user 0.028s sys 0.007s
I20240503 14:05:14.539144 7194 raft_consensus.cc:348] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.540120 7194 raft_consensus.cc:729] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:14.540843 7194 consensus_queue.cc:260] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.550127 6408 tablet_copy_source_session.cc:205] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:14.553920 7199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3586823a1bd546f08b8007d4b55ef43e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:14.567384 7194 ts_tablet_manager.cc:1398] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.031s user 0.008s sys 0.004s
I20240503 14:05:14.574568 6333 tablet_copy_service.cc:342] P 9a9ba562b78c4746b83a40ca7c30c00d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-a594e34044584aa09a50d15954cbb81b received from {username='slave'} at 127.0.0.1:59810
I20240503 14:05:14.575062 6333 tablet_copy_service.cc:433] P 9a9ba562b78c4746b83a40ca7c30c00d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-a594e34044584aa09a50d15954cbb81b on tablet a594e34044584aa09a50d15954cbb81b with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:14.589083 7199 tablet_copy_client.cc:783] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:14.589917 7199 tablet_copy_client.cc:647] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:14.594054 7199 tablet_copy_client.cc:523] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:14.600845 7194 ts_tablet_manager.cc:902] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:14.608736 7199 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
W20240503 14:05:14.643577 6289 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.195:35583, user_credentials={real_user=slave}} blocked reactor thread for 70663.8us
I20240503 14:05:14.658769 7199 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:14.659771 7199 tablet_bootstrap.cc:492] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:14.660526 7199 ts_tablet_manager.cc:1367] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.052s user 0.026s sys 0.008s
I20240503 14:05:14.663892 7199 raft_consensus.cc:348] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.664834 7199 raft_consensus.cc:729] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:14.665514 7199 consensus_queue.cc:260] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.667945 7199 ts_tablet_manager.cc:1398] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:05:14.671262 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-3586823a1bd546f08b8007d4b55ef43e received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:14.671723 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-3586823a1bd546f08b8007d4b55ef43e on tablet 3586823a1bd546f08b8007d4b55ef43e with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:14.722997 7194 tablet_copy_client.cc:308] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
I20240503 14:05:14.725095 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet ef212857f4874110a036926374b4d0cd from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:14.725764 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet ef212857f4874110a036926374b4d0cd from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-ef212857f4874110a036926374b4d0cd
I20240503 14:05:14.740415 6408 tablet_copy_source_session.cc:205] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:14.743340 7044 raft_consensus.cc:1203] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:14.748874 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef212857f4874110a036926374b4d0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:14.768296 7194 tablet_copy_client.cc:783] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:14.768996 7194 tablet_copy_client.cc:647] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:14.774142 7194 tablet_copy_client.cc:523] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:14.794243 7194 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:14.816392 7199 ts_tablet_manager.cc:902] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:14.833696 7194 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:14.834981 7194 tablet_bootstrap.cc:492] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:14.835943 7194 ts_tablet_manager.cc:1367] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.026s
I20240503 14:05:14.841245 7194 raft_consensus.cc:348] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.842468 7194 raft_consensus.cc:729] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:14.843375 7194 consensus_queue.cc:260] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.847151 7194 ts_tablet_manager.cc:1398] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240503 14:05:14.849929 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-ef212857f4874110a036926374b4d0cd received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:14.850386 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-ef212857f4874110a036926374b4d0cd on tablet ef212857f4874110a036926374b4d0cd with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:14.871659 7199 tablet_copy_client.cc:308] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.195:35583
I20240503 14:05:14.874620 6482 tablet_copy_service.cc:140] P 8e7d4e1e07e94f21b75ba42bf86334e1: Received BeginTabletCopySession request for tablet 861ddaea971348a596840c74fb1e8a71 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:40718)
I20240503 14:05:14.875245 6482 tablet_copy_service.cc:161] P 8e7d4e1e07e94f21b75ba42bf86334e1: Beginning new tablet copy session on tablet 861ddaea971348a596840c74fb1e8a71 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:40718: session id = 64a069ee90f74166b112f45b458364dc-861ddaea971348a596840c74fb1e8a71
I20240503 14:05:14.884399 6482 tablet_copy_source_session.cc:205] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:14.888283 7199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861ddaea971348a596840c74fb1e8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:14.904579 7199 tablet_copy_client.cc:783] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:14.905143 7199 tablet_copy_client.cc:647] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:14.908634 7199 tablet_copy_client.cc:523] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:14.917129 7199 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:14.944716 7199 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:14.945662 7199 tablet_bootstrap.cc:492] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:14.946285 7199 ts_tablet_manager.cc:1367] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.011s
I20240503 14:05:14.949736 7199 raft_consensus.cc:348] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.950850 7199 raft_consensus.cc:729] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:14.951768 7199 consensus_queue.cc:260] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:14.954973 7199 ts_tablet_manager.cc:1398] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240503 14:05:14.957442 6482 tablet_copy_service.cc:342] P 8e7d4e1e07e94f21b75ba42bf86334e1: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-861ddaea971348a596840c74fb1e8a71 received from {username='slave'} at 127.0.0.1:40718
I20240503 14:05:14.957971 6482 tablet_copy_service.cc:433] P 8e7d4e1e07e94f21b75ba42bf86334e1: ending tablet copy session 64a069ee90f74166b112f45b458364dc-861ddaea971348a596840c74fb1e8a71 on tablet 861ddaea971348a596840c74fb1e8a71 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:15.053799 7044 raft_consensus.cc:1203] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:15.139945 7044 raft_consensus.cc:1203] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:15.195561 7178 raft_consensus.cc:1050] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:15.200258 7178 consensus_queue.cc:237] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.206734 7044 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.206835 6468 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.208317 7105 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.210650 7211 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.214713 7043 raft_consensus.cc:1203] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:15.224040 6396 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.227427 7178 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.269304 7105 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.282455 6468 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.292589 6397 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.305835 7044 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.322849 6221 catalog_manager.cc:5484] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:15.361793 6322 consensus_queue.cc:237] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.367640 7044 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.368588 6472 raft_consensus.cc:1261] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.373596 7178 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:15.376032 7105 consensus_queue.cc:1035] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:15.390302 7211 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.392230 6468 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.400014 6206 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a594e34044584aa09a50d15954cbb81b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:15.401648 7044 raft_consensus.cc:2932] T a594e34044584aa09a50d15954cbb81b P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.413446 6218 catalog_manager.cc:5484] T a594e34044584aa09a50d15954cbb81b P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:15.428552 6386 tablet_service.cc:1514] Processing DeleteTablet for tablet a594e34044584aa09a50d15954cbb81b with delete_type TABLET_DATA_TOMBSTONED (TS 90a66da3b59e4511bdc1d35930fe558d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52860
I20240503 14:05:15.431016 7230 tablet_replica.cc:330] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: stopping tablet replica
I20240503 14:05:15.433094 7230 raft_consensus.cc:2229] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:15.434041 7230 raft_consensus.cc:2258] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:15.437583 7230 ts_tablet_manager.cc:1875] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:15.457072 7230 ts_tablet_manager.cc:1888] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:15.457778 7230 log.cc:1196] T a594e34044584aa09a50d15954cbb81b P 90a66da3b59e4511bdc1d35930fe558d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/wals/a594e34044584aa09a50d15954cbb81b
I20240503 14:05:15.459861 6205 catalog_manager.cc:4830] TS 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): tablet a594e34044584aa09a50d15954cbb81b (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:15.547917 7190 raft_consensus.cc:1050] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:15.550171 7190 consensus_queue.cc:237] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.558998 7044 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.559087 6468 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.559804 6322 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.561654 7221 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.562698 7222 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.564261 7190 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.593334 7190 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.601090 6468 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.601554 6322 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.606555 7044 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.613154 6219 catalog_manager.cc:5484] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:15.644110 6397 consensus_queue.cc:237] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.649914 7043 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.651868 6472 raft_consensus.cc:1261] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.653741 7220 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:15.655970 7221 consensus_queue.cc:1035] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:15.651783 7157 raft_consensus.cc:1050] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:15.662611 7190 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.663136 7157 consensus_queue.cc:237] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.668426 7193 raft_consensus.cc:1050] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:15.671000 7193 consensus_queue.cc:237] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.673714 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3586823a1bd546f08b8007d4b55ef43e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:15.674305 6472 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.680773 7044 raft_consensus.cc:2932] T 3586823a1bd546f08b8007d4b55ef43e P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.687247 6220 catalog_manager.cc:5484] T 3586823a1bd546f08b8007d4b55ef43e P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:15.689883 7043 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.689883 6468 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.691288 7193 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.692034 6322 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.692615 7190 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.696066 7220 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:15.724112 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet 3586823a1bd546f08b8007d4b55ef43e with delete_type TABLET_DATA_TOMBSTONED (TS 9a9ba562b78c4746b83a40ca7c30c00d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34752
I20240503 14:05:15.727840 7043 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.730199 6322 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.730324 7157 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:15.730998 6396 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:15.732409 7174 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:15.733347 7157 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:15.739038 7221 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.742595 6466 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.770637 6220 catalog_manager.cc:5484] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:15.797784 6317 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.812652 7044 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.817389 7241 tablet_replica.cc:330] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: stopping tablet replica
I20240503 14:05:15.818593 7241 raft_consensus.cc:2229] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:15.819365 7241 raft_consensus.cc:2258] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:15.832119 7245 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.844962 7241 ts_tablet_manager.cc:1875] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:15.853106 6393 consensus_queue.cc:237] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
W20240503 14:05:15.853385 6436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.250.193:37155, user_credentials={real_user=slave}} blocked reactor thread for 60541.7us
I20240503 14:05:15.852851 6396 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.866984 7241 ts_tablet_manager.cc:1888] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:15.867538 7241 log.cc:1196] T 3586823a1bd546f08b8007d4b55ef43e P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/wals/3586823a1bd546f08b8007d4b55ef43e
I20240503 14:05:15.869284 6206 catalog_manager.cc:4830] TS 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): tablet 3586823a1bd546f08b8007d4b55ef43e (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:15.859196 7043 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.878073 7044 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.880640 7222 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:15.880404 6218 catalog_manager.cc:5484] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:15.887509 7221 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.894456 7044 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.900207 6321 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.906106 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef212857f4874110a036926374b4d0cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:15.913331 6317 raft_consensus.cc:1261] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.920616 7220 consensus_queue.cc:1035] T ef212857f4874110a036926374b4d0cd P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:15.920322 6219 catalog_manager.cc:5484] T ef212857f4874110a036926374b4d0cd P 64a069ee90f74166b112f45b458364dc reported cstate change: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.930328 6317 raft_consensus.cc:2932] T ef212857f4874110a036926374b4d0cd P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.946439 6455 tablet_service.cc:1514] Processing DeleteTablet for tablet ef212857f4874110a036926374b4d0cd with delete_type TABLET_DATA_TOMBSTONED (TS 8e7d4e1e07e94f21b75ba42bf86334e1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58394
I20240503 14:05:15.947256 6466 consensus_queue.cc:237] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:15.953991 7248 tablet_replica.cc:330] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: stopping tablet replica
I20240503 14:05:15.955235 6396 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.955281 7248 raft_consensus.cc:2229] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:15.955797 7044 raft_consensus.cc:1261] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:15.956336 7248 raft_consensus.cc:2258] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:15.957280 7246 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:15.958381 7157 consensus_queue.cc:1035] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:15.960659 7248 ts_tablet_manager.cc:1875] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:15.966974 7171 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.971918 6396 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.982129 7044 raft_consensus.cc:2932] T 861ddaea971348a596840c74fb1e8a71 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:15.984177 7248 ts_tablet_manager.cc:1888] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:15.984654 7248 log.cc:1196] T ef212857f4874110a036926374b4d0cd P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/wals/ef212857f4874110a036926374b4d0cd
I20240503 14:05:15.995841 6219 catalog_manager.cc:5484] T 861ddaea971348a596840c74fb1e8a71 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:16.012617 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet 861ddaea971348a596840c74fb1e8a71 with delete_type TABLET_DATA_TOMBSTONED (TS 9a9ba562b78c4746b83a40ca7c30c00d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34752
I20240503 14:05:16.013595 7241 tablet_replica.cc:330] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: stopping tablet replica
I20240503 14:05:16.018980 6208 catalog_manager.cc:4830] TS 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): tablet ef212857f4874110a036926374b4d0cd (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:16.022843 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 861ddaea971348a596840c74fb1e8a71 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:16.024672 7241 raft_consensus.cc:2229] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:16.025375 7241 raft_consensus.cc:2258] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:16.028486 7241 ts_tablet_manager.cc:1875] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:16.043134 7241 ts_tablet_manager.cc:1888] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:16.043462 7241 log.cc:1196] T 861ddaea971348a596840c74fb1e8a71 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/wals/861ddaea971348a596840c74fb1e8a71
I20240503 14:05:16.044698 6206 catalog_manager.cc:4830] TS 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): tablet 861ddaea971348a596840c74fb1e8a71 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:17.099961 6472 consensus_queue.cc:237] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.107365 6396 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.109076 6322 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.109083 7252 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:17.112026 7252 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:05:17.113844 6438 consensus_peers.cc:485] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56476ec0a3874a83bc22ec0c101088c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:17.122694 7251 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.124346 6322 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.125365 6396 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.144677 6219 catalog_manager.cc:5484] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:17.151422 6397 consensus_queue.cc:237] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
W20240503 14:05:17.163148 6364 consensus_peers.cc:485] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e388b249118b41faa4cb6efffb8fecea. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:17.163161 6468 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.164333 6322 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.165436 7193 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:05:17.166549 7221 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:17.204753 7193 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.205924 6322 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.221587 6396 consensus_queue.cc:237] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.223407 6468 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.246322 6468 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 14:05:17.246994 6364 consensus_peers.cc:485] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf848b599cbc45ecb8772ecfb50276e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:17.245503 6219 catalog_manager.cc:5484] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.247879 7259 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:17.250587 6322 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:05:17.253007 7259 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:17.266777 6468 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.264935 7265 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.287955 6321 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.306461 6221 catalog_manager.cc:5484] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 2 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.306777 6321 consensus_queue.cc:237] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.316200 6396 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.316999 6472 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:17.317555 7211 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:05:17.318533 6289 consensus_peers.cc:485] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6845d794633f4f47a07ae27a88362ffa. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:17.319875 7211 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:17.329182 6396 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.327406 7249 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.340370 6472 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.350229 6221 catalog_manager.cc:5484] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:17.557503 7275 ts_tablet_manager.cc:902] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:17.563192 7275 tablet_copy_client.cc:308] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
I20240503 14:05:17.564939 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet e388b249118b41faa4cb6efffb8fecea from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:17.565568 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet e388b249118b41faa4cb6efffb8fecea from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-e388b249118b41faa4cb6efffb8fecea
I20240503 14:05:17.572453 6408 tablet_copy_source_session.cc:205] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:17.575809 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388b249118b41faa4cb6efffb8fecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:17.594588 7275 tablet_copy_client.cc:783] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:17.595290 7275 tablet_copy_client.cc:647] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:17.599282 7275 tablet_copy_client.cc:523] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:17.607327 7275 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:17.629596 7275 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:17.630472 7275 tablet_bootstrap.cc:492] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:17.631150 7275 ts_tablet_manager.cc:1367] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.008s
I20240503 14:05:17.633720 7275 raft_consensus.cc:348] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.634481 7275 raft_consensus.cc:729] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:17.635013 7275 consensus_queue.cc:260] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.638120 7275 ts_tablet_manager.cc:1398] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240503 14:05:17.640266 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-e388b249118b41faa4cb6efffb8fecea received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:17.640704 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-e388b249118b41faa4cb6efffb8fecea on tablet e388b249118b41faa4cb6efffb8fecea with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:17.672220 7275 ts_tablet_manager.cc:902] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
I20240503 14:05:17.678751 7275 tablet_copy_client.cc:308] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.195:35583
I20240503 14:05:17.680675 6482 tablet_copy_service.cc:140] P 8e7d4e1e07e94f21b75ba42bf86334e1: Received BeginTabletCopySession request for tablet 56476ec0a3874a83bc22ec0c101088c8 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:40718)
I20240503 14:05:17.681267 6482 tablet_copy_service.cc:161] P 8e7d4e1e07e94f21b75ba42bf86334e1: Beginning new tablet copy session on tablet 56476ec0a3874a83bc22ec0c101088c8 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:40718: session id = 64a069ee90f74166b112f45b458364dc-56476ec0a3874a83bc22ec0c101088c8
I20240503 14:05:17.691982 6482 tablet_copy_source_session.cc:205] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:17.694806 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56476ec0a3874a83bc22ec0c101088c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:17.707679 7275 tablet_copy_client.cc:783] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:17.708268 7275 tablet_copy_client.cc:647] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:17.713209 7275 tablet_copy_client.cc:523] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:17.720688 7275 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:17.738580 7275 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:17.739328 7275 tablet_bootstrap.cc:492] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:17.739946 7275 ts_tablet_manager.cc:1367] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240503 14:05:17.741029 7287 ts_tablet_manager.cc:902] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097)
I20240503 14:05:17.742542 7275 raft_consensus.cc:348] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.743474 7275 raft_consensus.cc:729] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:17.744141 7275 consensus_queue.cc:260] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.755499 7275 ts_tablet_manager.cc:1398] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:05:17.764101 6482 tablet_copy_service.cc:342] P 8e7d4e1e07e94f21b75ba42bf86334e1: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-56476ec0a3874a83bc22ec0c101088c8 received from {username='slave'} at 127.0.0.1:40718
I20240503 14:05:17.764570 6482 tablet_copy_service.cc:433] P 8e7d4e1e07e94f21b75ba42bf86334e1: ending tablet copy session 64a069ee90f74166b112f45b458364dc-56476ec0a3874a83bc22ec0c101088c8 on tablet 56476ec0a3874a83bc22ec0c101088c8 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:17.779026 7287 tablet_copy_client.cc:308] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.194:39097
I20240503 14:05:17.780827 6408 tablet_copy_service.cc:140] P 90a66da3b59e4511bdc1d35930fe558d: Received BeginTabletCopySession request for tablet bf848b599cbc45ecb8772ecfb50276e7 from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:57966)
I20240503 14:05:17.781396 6408 tablet_copy_service.cc:161] P 90a66da3b59e4511bdc1d35930fe558d: Beginning new tablet copy session on tablet bf848b599cbc45ecb8772ecfb50276e7 from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:57966: session id = 64a069ee90f74166b112f45b458364dc-bf848b599cbc45ecb8772ecfb50276e7
I20240503 14:05:17.789709 6408 tablet_copy_source_session.cc:205] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:17.792405 7287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf848b599cbc45ecb8772ecfb50276e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:17.808651 7287 tablet_copy_client.cc:783] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:17.809303 7287 tablet_copy_client.cc:647] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:17.813916 7287 tablet_copy_client.cc:523] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:17.823024 7275 ts_tablet_manager.cc:902] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155)
I20240503 14:05:17.823801 7287 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:17.849133 7275 tablet_copy_client.cc:308] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: tablet copy: Beginning tablet copy session from remote peer at address 127.4.250.193:37155
I20240503 14:05:17.851128 7287 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:17.852128 7287 tablet_bootstrap.cc:492] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:17.852856 7287 ts_tablet_manager.cc:1367] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20240503 14:05:17.851317 6333 tablet_copy_service.cc:140] P 9a9ba562b78c4746b83a40ca7c30c00d: Received BeginTabletCopySession request for tablet 6845d794633f4f47a07ae27a88362ffa from peer 64a069ee90f74166b112f45b458364dc ({username='slave'} at 127.0.0.1:59810)
I20240503 14:05:17.855497 6333 tablet_copy_service.cc:161] P 9a9ba562b78c4746b83a40ca7c30c00d: Beginning new tablet copy session on tablet 6845d794633f4f47a07ae27a88362ffa from peer 64a069ee90f74166b112f45b458364dc at {username='slave'} at 127.0.0.1:59810: session id = 64a069ee90f74166b112f45b458364dc-6845d794633f4f47a07ae27a88362ffa
I20240503 14:05:17.857049 7287 raft_consensus.cc:348] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.858039 7287 raft_consensus.cc:729] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:17.858686 7287 consensus_queue.cc:260] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.865391 7287 ts_tablet_manager.cc:1398] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240503 14:05:17.867877 6333 tablet_copy_source_session.cc:205] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:05:17.870371 6408 tablet_copy_service.cc:342] P 90a66da3b59e4511bdc1d35930fe558d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-bf848b599cbc45ecb8772ecfb50276e7 received from {username='slave'} at 127.0.0.1:57966
I20240503 14:05:17.870860 6408 tablet_copy_service.cc:433] P 90a66da3b59e4511bdc1d35930fe558d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-bf848b599cbc45ecb8772ecfb50276e7 on tablet bf848b599cbc45ecb8772ecfb50276e7 with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:17.871363 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6845d794633f4f47a07ae27a88362ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:05:17.892772 7275 tablet_copy_client.cc:783] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 0 data blocks...
I20240503 14:05:17.893386 7275 tablet_copy_client.cc:647] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:05:17.901068 7275 tablet_copy_client.cc:523] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:05:17.913108 7275 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Bootstrap starting.
I20240503 14:05:17.930617 7275 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:05:17.931504 7275 tablet_bootstrap.cc:492] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Bootstrap complete.
I20240503 14:05:17.932233 7275 ts_tablet_manager.cc:1367] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.001s
I20240503 14:05:17.934634 7275 raft_consensus.cc:348] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.935420 7275 raft_consensus.cc:729] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64a069ee90f74166b112f45b458364dc, State: Initialized, Role: LEARNER
I20240503 14:05:17.935900 7275 consensus_queue.cc:260] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:17.937695 7275 ts_tablet_manager.cc:1398] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:05:17.939275 6333 tablet_copy_service.cc:342] P 9a9ba562b78c4746b83a40ca7c30c00d: Request end of tablet copy session 64a069ee90f74166b112f45b458364dc-6845d794633f4f47a07ae27a88362ffa received from {username='slave'} at 127.0.0.1:59810
I20240503 14:05:17.939692 6333 tablet_copy_service.cc:433] P 9a9ba562b78c4746b83a40ca7c30c00d: ending tablet copy session 64a069ee90f74166b112f45b458364dc-6845d794633f4f47a07ae27a88362ffa on tablet 6845d794633f4f47a07ae27a88362ffa with peer 64a069ee90f74166b112f45b458364dc
I20240503 14:05:18.029806 7044 raft_consensus.cc:1203] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:18.286862 7044 raft_consensus.cc:1203] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:18.313699 7044 raft_consensus.cc:1203] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 14:05:18.324484 7044 raft_consensus.cc:1203] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:05:18.571533 7193 raft_consensus.cc:1050] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:18.573781 7193 consensus_queue.cc:237] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.580955 7044 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.581593 6472 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.583165 7259 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:18.584451 7300 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.587149 6321 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.588629 7300 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.609957 7259 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.613698 6321 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.616621 6472 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.632479 6221 catalog_manager.cc:5484] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.632750 7044 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.666193 6396 consensus_queue.cc:237] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.670452 7044 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:18.672006 6321 raft_consensus.cc:1261] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:18.673166 7193 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:18.673952 7259 consensus_queue.cc:1035] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:18.680683 7259 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 90a66da3b59e4511bdc1d35930fe558d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.688242 6321 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.693768 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e388b249118b41faa4cb6efffb8fecea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:18.698632 7044 raft_consensus.cc:2932] T e388b249118b41faa4cb6efffb8fecea P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.714495 6219 catalog_manager.cc:5484] T e388b249118b41faa4cb6efffb8fecea P 9a9ba562b78c4746b83a40ca7c30c00d reported cstate change: config changed from index 3 to 4, VOTER 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195) evicted. New cstate: current_term: 1 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.729570 6455 tablet_service.cc:1514] Processing DeleteTablet for tablet e388b249118b41faa4cb6efffb8fecea with delete_type TABLET_DATA_TOMBSTONED (TS 8e7d4e1e07e94f21b75ba42bf86334e1 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58394
I20240503 14:05:18.756868 7311 tablet_replica.cc:330] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: stopping tablet replica
I20240503 14:05:18.758139 7311 raft_consensus.cc:2229] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:18.759047 7311 raft_consensus.cc:2258] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:18.767261 7311 ts_tablet_manager.cc:1875] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:18.781100 7259 raft_consensus.cc:1050] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:18.783917 7259 consensus_queue.cc:237] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.787086 7311 ts_tablet_manager.cc:1888] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:18.787622 7311 log.cc:1196] T e388b249118b41faa4cb6efffb8fecea P 8e7d4e1e07e94f21b75ba42bf86334e1: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-2-root/wals/e388b249118b41faa4cb6efffb8fecea
I20240503 14:05:18.790516 6208 catalog_manager.cc:4830] TS 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583): tablet e388b249118b41faa4cb6efffb8fecea (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:18.791708 6466 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:05:18.793290 6322 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:05:18.795075 7259 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.797120 7193 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.791898 7044 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 LEARNER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:05:18.800451 7300 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.802364 7249 raft_consensus.cc:1050] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:18.804988 7249 consensus_queue.cc:237] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.817391 7265 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.825645 6466 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.831861 7045 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.832217 6468 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.835676 7312 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:18.838048 6396 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.843541 7312 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240503 14:05:18.845321 6322 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.846740 7249 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:18.846583 7043 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.849725 7171 raft_consensus.cc:1050] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1: attempting to promote NON_VOTER 64a069ee90f74166b112f45b458364dc to VOTER
I20240503 14:05:18.857782 7171 consensus_queue.cc:237] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.859181 6220 catalog_manager.cc:5484] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.900461 7045 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 LEARNER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.900671 6395 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.902828 7309 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:18.903175 6321 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:05:18.909021 7249 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.910496 6393 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.912353 7308 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:05:18.915320 7171 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:05:18.928859 7298 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.931198 6395 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.946574 7045 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.939522 6321 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.949442 6220 catalog_manager.cc:5484] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:18.962534 7043 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.988142 6468 consensus_queue.cc:237] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:18.989431 6472 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:18.992890 6219 catalog_manager.cc:5484] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 2 to 3, 64a069ee90f74166b112f45b458364dc (127.4.250.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.017657 6321 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:19.025826 7252 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:19.030378 7044 raft_consensus.cc:1261] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:19.034119 7252 consensus_queue.cc:1035] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:19.040300 7308 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.042294 6321 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.054121 6316 consensus_queue.cc:237] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:19.061079 7042 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:19.059067 7044 raft_consensus.cc:2932] T 56476ec0a3874a83bc22ec0c101088c8 P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.061319 6466 raft_consensus.cc:1261] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:05:19.062450 7310 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:05:19.064327 7249 consensus_queue.cc:1035] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:19.065873 6208 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56476ec0a3874a83bc22ec0c101088c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:19.068804 6395 consensus_queue.cc:237] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } }
I20240503 14:05:19.082137 6218 catalog_manager.cc:5484] T 56476ec0a3874a83bc22ec0c101088c8 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:19.084786 7045 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:05:19.086688 6466 raft_consensus.cc:1261] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Refusing update from remote peer 90a66da3b59e4511bdc1d35930fe558d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:05:19.089033 7249 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.090339 7193 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:19.091967 7259 consensus_queue.cc:1035] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:05:19.097811 6386 tablet_service.cc:1514] Processing DeleteTablet for tablet 56476ec0a3874a83bc22ec0c101088c8 with delete_type TABLET_DATA_TOMBSTONED (TS 90a66da3b59e4511bdc1d35930fe558d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52860
I20240503 14:05:19.101814 7193 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.100158 7045 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.112282 6206 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6845d794633f4f47a07ae27a88362ffa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:19.105352 6466 raft_consensus.cc:2932] T 6845d794633f4f47a07ae27a88362ffa P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.122821 7325 tablet_replica.cc:330] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: stopping tablet replica
I20240503 14:05:19.123898 7325 raft_consensus.cc:2229] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:19.124791 7325 raft_consensus.cc:2258] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:19.104158 7042 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 64a069ee90f74166b112f45b458364dc [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.127952 6205 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf848b599cbc45ecb8772ecfb50276e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:05:19.128852 7325 ts_tablet_manager.cc:1875] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:19.151494 6468 raft_consensus.cc:2932] T bf848b599cbc45ecb8772ecfb50276e7 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.149421 6218 catalog_manager.cc:5484] T bf848b599cbc45ecb8772ecfb50276e7 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index 3 to 4, VOTER 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193) evicted. New cstate: current_term: 2 leader_uuid: "90a66da3b59e4511bdc1d35930fe558d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:05:19.173112 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet bf848b599cbc45ecb8772ecfb50276e7 with delete_type TABLET_DATA_TOMBSTONED (TS 9a9ba562b78c4746b83a40ca7c30c00d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34752
I20240503 14:05:19.163466 6219 catalog_manager.cc:5484] T 6845d794633f4f47a07ae27a88362ffa P 64a069ee90f74166b112f45b458364dc reported cstate change: config changed from index 3 to 4, VOTER 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194) evicted. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: false } } }
I20240503 14:05:19.199153 7325 ts_tablet_manager.cc:1888] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:19.199733 7325 log.cc:1196] T 56476ec0a3874a83bc22ec0c101088c8 P 90a66da3b59e4511bdc1d35930fe558d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/wals/56476ec0a3874a83bc22ec0c101088c8
I20240503 14:05:19.201736 6205 catalog_manager.cc:4830] TS 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): tablet 56476ec0a3874a83bc22ec0c101088c8 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:19.204105 6386 tablet_service.cc:1514] Processing DeleteTablet for tablet 6845d794633f4f47a07ae27a88362ffa with delete_type TABLET_DATA_TOMBSTONED (TS 90a66da3b59e4511bdc1d35930fe558d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52860
I20240503 14:05:19.205224 7325 tablet_replica.cc:330] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: stopping tablet replica
I20240503 14:05:19.206501 7325 raft_consensus.cc:2229] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:19.207334 7325 raft_consensus.cc:2258] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:19.208577 7327 tablet_replica.cc:330] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: stopping tablet replica
I20240503 14:05:19.210364 7327 raft_consensus.cc:2229] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:05:19.211284 7327 raft_consensus.cc:2258] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:05:19.212587 7325 ts_tablet_manager.cc:1875] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:19.226824 7327 ts_tablet_manager.cc:1875] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:05:19.233636 7325 ts_tablet_manager.cc:1888] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:05:19.234380 7325 log.cc:1196] T 6845d794633f4f47a07ae27a88362ffa P 90a66da3b59e4511bdc1d35930fe558d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-1-root/wals/6845d794633f4f47a07ae27a88362ffa
I20240503 14:05:19.236285 6205 catalog_manager.cc:4830] TS 90a66da3b59e4511bdc1d35930fe558d (127.4.250.194:39097): tablet 6845d794633f4f47a07ae27a88362ffa (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:19.252049 7327 ts_tablet_manager.cc:1888] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:05:19.253875 7327 log.cc:1196] T bf848b599cbc45ecb8772ecfb50276e7 P 9a9ba562b78c4746b83a40ca7c30c00d: Deleting WAL directory at /tmp/dist-test-task_0y0Qd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714745034190430-5099-0/minicluster-data/ts-0-root/wals/bf848b599cbc45ecb8772ecfb50276e7
I20240503 14:05:19.258646 6206 catalog_manager.cc:4830] TS 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155): tablet bf848b599cbc45ecb8772ecfb50276e7 (table test-workload [id=5962ac28e0864fefbed28c4368b99536]) successfully deleted
I20240503 14:05:20.210196 6316 consensus_queue.cc:237] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:20.216542 6396 raft_consensus.cc:1261] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:05:20.216845 6289 consensus_peers.cc:485] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7257be676307482195052c59227129f0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:20.216583 6468 raft_consensus.cc:1261] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Refusing update from remote peer 9a9ba562b78c4746b83a40ca7c30c00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.218410 7330 consensus_queue.cc:1035] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.219190 7312 consensus_queue.cc:1035] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.230712 7312 raft_consensus.cc:2932] T 7257be676307482195052c59227129f0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.234400 6395 raft_consensus.cc:2932] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.245172 6468 raft_consensus.cc:2932] T 7257be676307482195052c59227129f0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.255097 6219 catalog_manager.cc:5484] T 7257be676307482195052c59227129f0 P 90a66da3b59e4511bdc1d35930fe558d reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.266117 6472 consensus_queue.cc:237] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:20.298705 6321 raft_consensus.cc:1261] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:05:20.299677 6438 consensus_peers.cc:485] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df77168e3be74202b705c73539c514d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:20.303529 7298 consensus_queue.cc:1035] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.311417 6395 raft_consensus.cc:1261] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.315346 7308 consensus_queue.cc:1035] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.321560 7329 raft_consensus.cc:2932] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.323680 6316 raft_consensus.cc:2932] T df77168e3be74202b705c73539c514d0 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.346737 6472 consensus_queue.cc:237] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:20.349414 6396 raft_consensus.cc:2932] T df77168e3be74202b705c73539c514d0 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
W20240503 14:05:20.359197 6438 consensus_peers.cc:485] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5ce484e160849488b2637359d07ba88. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:20.359505 6221 catalog_manager.cc:5484] T df77168e3be74202b705c73539c514d0 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:20.364655 6393 raft_consensus.cc:1261] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.366904 7298 consensus_queue.cc:1035] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.367867 6316 raft_consensus.cc:1261] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.372898 7298 consensus_queue.cc:1035] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.383229 7345 raft_consensus.cc:2932] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.386869 6396 raft_consensus.cc:2932] T f5ce484e160849488b2637359d07ba88 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.400117 6472 consensus_queue.cc:237] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } }
I20240503 14:05:20.404294 6220 catalog_manager.cc:5484] T f5ce484e160849488b2637359d07ba88 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:05:20.413516 6438 consensus_peers.cc:485] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 -> Peer 64a069ee90f74166b112f45b458364dc (127.4.250.196:46183): Couldn't send request to peer 64a069ee90f74166b112f45b458364dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f379675884943d6b074d7af75fdbf64. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:05:20.412989 6316 raft_consensus.cc:2932] T f5ce484e160849488b2637359d07ba88 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } attrs { replace: true } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.413087 6396 raft_consensus.cc:1261] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.417649 6321 raft_consensus.cc:1261] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Refusing update from remote peer 8e7d4e1e07e94f21b75ba42bf86334e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:05:20.418463 7298 consensus_queue.cc:1035] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.420203 7345 consensus_queue.cc:1035] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:05:20.444988 7329 raft_consensus.cc:2932] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.446666 6316 raft_consensus.cc:2932] T 4f379675884943d6b074d7af75fdbf64 P 9a9ba562b78c4746b83a40ca7c30c00d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.450824 6396 raft_consensus.cc:2932] T 4f379675884943d6b074d7af75fdbf64 P 90a66da3b59e4511bdc1d35930fe558d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } } }
I20240503 14:05:20.461127 6221 catalog_manager.cc:5484] T 4f379675884943d6b074d7af75fdbf64 P 8e7d4e1e07e94f21b75ba42bf86334e1 reported cstate change: config changed from index -1 to 2, NON_VOTER 64a069ee90f74166b112f45b458364dc (127.4.250.196) added. New cstate: current_term: 1 leader_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9a9ba562b78c4746b83a40ca7c30c00d" member_type: VOTER last_known_addr { host: "127.4.250.193" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90a66da3b59e4511bdc1d35930fe558d" member_type: VOTER last_known_addr { host: "127.4.250.194" port: 39097 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e7d4e1e07e94f21b75ba42bf86334e1" member_type: VOTER last_known_addr { host: "127.4.250.195" port: 35583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a069ee90f74166b112f45b458364dc" member_type: NON_VOTER last_known_addr { host: "127.4.250.196" port: 46183 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:05:20.619755 7351 ts_tablet_manager.cc:902] T 7257be676307482195052c59227129f0 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 9a9ba562b78c4746b83a40ca7c30c00d (127.4.250.193:37155)
F20240503 14:05:20.620847 7351 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.queue_time_us, tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.queue_time_us) (tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.run_cpu_time_us, tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.run_cpu_time_us) (tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.run_wall_time_us, tablet-download-pool-3586823a1bd546f08b8007d4b55ef43e.run_wall_time_us) (tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.queue_time_us, tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.queue_time_us) (tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.run_cpu_time_us, tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.run_cpu_time_us) (tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.run_wall_time_us, tablet-download-pool-56476ec0a3874a83bc22ec0c101088c8.run_wall_time_us) (tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.queue_time_us, tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.queue_time_us) (tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.run_cpu_time_us, tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.run_cpu_time_us) (tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.run_wall_time_us, tablet-download-pool-665b49a99fb74a1ab8ad655e76ca0e34.run_wall_time_us) (tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.queue_time_us, tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.queue_time_us) (tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.run_cpu_time_us, tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.run_cpu_time_us) (tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.run_wall_time_us, tablet-download-pool-6845d794633f4f47a07ae27a88362ffa.run_wall_time_us) (tablet-download-pool-7257be676307482195052c59227129f0.queue_time_us, tablet-download-pool-7257be676307482195052c59227129f0.queue_time_us) (tablet-download-pool-861ddaea971348a596840c74fb1e8a71.queue_time_us, tablet-download-pool-861ddaea971348a596840c74fb1e8a71.queue_time_us) (tablet-download-pool-861ddaea971348a596840c74fb1e8a71.run_cpu_time_us, tablet-download-pool-861ddaea971348a596840c74fb1e8a71.run_cpu_time_us) (tablet-download-pool-861ddaea971348a596840c74fb1e8a71.run_wall_time_us, tablet-download-pool-861ddaea971348a596840c74fb1e8a71.run_wall_time_us) (tablet-download-pool-a19868aca07442c4bc78952934029bb9.queue_time_us, tablet-download-pool-a19868aca07442c4bc78952934029bb9.queue_time_us) (tablet-download-pool-a19868aca07442c4bc78952934029bb9.run_cpu_time_us, tablet-download-pool-a19868aca07442c4bc78952934029bb9.run_cpu_time_us) (tablet-download-pool-a19868aca07442c4bc78952934029bb9.run_wall_time_us, tablet-download-pool-a19868aca07442c4bc78952934029bb9.run_wall_time_us) (tablet-download-pool-a594e34044584aa09a50d15954cbb81b.queue_time_us, tablet-download-pool-a594e34044584aa09a50d15954cbb81b.queue_time_us) (tablet-download-pool-a594e34044584aa09a50d15954cbb81b.run_cpu_time_us, tablet-download-pool-a594e34044584aa09a50d15954cbb81b.run_cpu_time_us) (tablet-download-pool-a594e34044584aa09a50d15954cbb81b.run_wall_time_us, tablet-download-pool-a594e34044584aa09a50d15954cbb81b.run_wall_time_us) (tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.queue_time_us, tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.queue_time_us) (tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.run_cpu_time_us, tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.run_cpu_time_us) (tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.run_wall_time_us, tablet-download-pool-a8cbd74268fe4e5d8dc5e5d3cb1ed6d1.run_wall_time_us) (tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.queue_time_us, tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.queue_time_us) (tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.run_cpu_time_us, tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.run_cpu_time_us) (tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.run_wall_time_us, tablet-download-pool-b0285fa210834b8c9b116d0cf7bb2430.run_wall_time_us) (tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.queue_time_us, tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.queue_time_us) (tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.run_cpu_time_us, tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.run_cpu_time_us) (tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.run_wall_time_us, tablet-download-pool-bf848b599cbc45ecb8772ecfb50276e7.run_wall_time_us) (tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.queue_time_us, tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.queue_time_us) (tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.run_cpu_time_us, tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.run_cpu_time_us) (tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.run_wall_time_us, tablet-download-pool-e388b249118b41faa4cb6efffb8fecea.run_wall_time_us) (tablet-download-pool-ef212857f4874110a036926374b4d0cd.queue_time_us, tablet-download-pool-ef212857f4874110a036926374b4d0cd.queue_time_us) (tablet-download-pool-ef212857f4874110a036926374b4d0cd.run_cpu_time_us, tablet-download-pool-ef212857f4874110a036926374b4d0cd.run_cpu_time_us) (tablet-download-pool-ef212857f4874110a036926374b4d0cd.run_wall_time_us, tablet-download-pool-ef212857f4874110a036926374b4d0cd.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714745120 (unix time) try "date -d @1714745120" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000013eb) received by PID 5099 (TID 0x7ff6983c4700) from PID 5099; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7ff76797f980 (unknown) at ??:0
@ 0x7ff760ef3fb7 gsignal at ??:0
I20240503 14:05:20.726969 7354 ts_tablet_manager.cc:902] T df77168e3be74202b705c73539c514d0 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
@ 0x7ff760ef5921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7ff763ce9da4 google::LogMessage::SendToLog() at ??:0
@ 0x7ff763cea950 google::LogMessage::Flush() at ??:0
@ 0x7ff763cefa8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff7646773c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7ff76464d94f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7ff76464b0a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:05:20.895824 7365 ts_tablet_manager.cc:902] T 4f379675884943d6b074d7af75fdbf64 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
@ 0x7ff76c46b422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:05:20.906230 7367 ts_tablet_manager.cc:902] T f5ce484e160849488b2637359d07ba88 P 64a069ee90f74166b112f45b458364dc: Initiating tablet copy from peer 8e7d4e1e07e94f21b75ba42bf86334e1 (127.4.250.195:35583)
@ 0x7ff76c471c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ff76c5074ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ff76c5383cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ff76c51e1ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff76c51e17a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7ff76c51e10a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7ff76c51e0d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7ff76c51d2fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7ff76d8c3da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7ff76d8c3bda std::__1::function<>::operator()() at ??:0
@ 0x7ff76464f237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff764652692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff76465264a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7ff7646525da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7ff7646525a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7ff76465189e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7ff76d8c3da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7ff76d8c3bda std::__1::function<>::operator()() at ??:0
@ 0x7ff764631937 kudu::Thread::SuperviseThread() at ??:0