[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:55:11.314807 23834 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.70.190:42367
I20240501 13:55:11.316277 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:55:11.317471 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:11.338181 23843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:11.338866 23841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:11.339840 23840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:11.340646 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:12.467371 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:12.467649 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:12.467868 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571712467844 us; error 0 us; skew 500 ppm
I20240501 13:55:12.468770 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:12.478928 23834 webserver.cc:460] Webserver started at http://127.23.70.190:42873/ using document root <none> and password file <none>
I20240501 13:55:12.479869 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:12.480087 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:12.480566 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:12.487926 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/master-0-root/instance:
uuid: "9f574cbd1a4d41f3985dcb9508e980db"
format_stamp: "Formatted at 2024-05-01 13:55:12 on dist-test-slave-s72h"
I20240501 13:55:12.496479 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20240501 13:55:12.502050 23849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:12.503268 23834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:55:12.503647 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/master-0-root
uuid: "9f574cbd1a4d41f3985dcb9508e980db"
format_stamp: "Formatted at 2024-05-01 13:55:12 on dist-test-slave-s72h"
I20240501 13:55:12.503991 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:12.563925 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:12.565523 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:55:12.566017 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:12.653545 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.190:42367
I20240501 13:55:12.653631 23900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.190:42367 every 8 connection(s)
I20240501 13:55:12.662082 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:12.690402 23901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: Bootstrap starting.
I20240501 13:55:12.696597 23901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:12.698491 23901 log.cc:824] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: Log is configured to *not* fsync() on all Append() calls
I20240501 13:55:12.703586 23901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: No bootstrap required, opened a new log
I20240501 13:55:12.728288 23901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER }
I20240501 13:55:12.728953 23901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:12.729208 23901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f574cbd1a4d41f3985dcb9508e980db, State: Initialized, Role: FOLLOWER
I20240501 13:55:12.730031 23901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER }
I20240501 13:55:12.730533 23901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:55:12.730815 23901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:55:12.731142 23901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:12.735248 23901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER }
I20240501 13:55:12.735991 23901 leader_election.cc:302] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f574cbd1a4d41f3985dcb9508e980db; no voters:
I20240501 13:55:12.738130 23901 leader_election.cc:288] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:55:12.738507 23904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:12.741576 23904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 1 LEADER]: Becoming Leader. State: Replica: 9f574cbd1a4d41f3985dcb9508e980db, State: Running, Role: LEADER
I20240501 13:55:12.742918 23904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER }
I20240501 13:55:12.743886 23901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:55:12.756675 23906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f574cbd1a4d41f3985dcb9508e980db. Latest consensus state: current_term: 1 leader_uuid: "9f574cbd1a4d41f3985dcb9508e980db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER } }
I20240501 13:55:12.756731 23905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f574cbd1a4d41f3985dcb9508e980db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f574cbd1a4d41f3985dcb9508e980db" member_type: VOTER } }
I20240501 13:55:12.757643 23906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [sys.catalog]: This master's current role is: LEADER
I20240501 13:55:12.757633 23905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [sys.catalog]: This master's current role is: LEADER
I20240501 13:55:12.761721 23913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:55:12.776778 23913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:55:12.778666 23834 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:55:12.796917 23913 catalog_manager.cc:1347] Generated new cluster ID: 82dc66e96d744cfba383d8a60ef2dde4
I20240501 13:55:12.797263 23913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:55:12.842432 23913 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:55:12.844442 23913 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:55:12.858282 23913 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: Generated new TSK 0
I20240501 13:55:12.859313 23913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:55:12.910487 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:12.922057 23923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:12.924392 23924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:12.926627 23926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:12.928074 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:12.929041 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:12.929327 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:12.929536 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571712929518 us; error 0 us; skew 500 ppm
I20240501 13:55:12.930459 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:12.935410 23834 webserver.cc:460] Webserver started at http://127.23.70.129:39061/ using document root <none> and password file <none>
I20240501 13:55:12.936143 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:12.936414 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:12.936801 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:12.938369 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-0-root/instance:
uuid: "093b61249a29465596a6542c3b6da169"
format_stamp: "Formatted at 2024-05-01 13:55:12 on dist-test-slave-s72h"
I20240501 13:55:12.945621 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20240501 13:55:12.951596 23931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:12.952531 23834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240501 13:55:12.952845 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
uuid: "093b61249a29465596a6542c3b6da169"
format_stamp: "Formatted at 2024-05-01 13:55:12 on dist-test-slave-s72h"
I20240501 13:55:12.953141 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:12.986572 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:12.988045 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:12.990924 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:12.996623 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:12.996894 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:12.997176 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:12.997360 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.112334 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.129:35903
I20240501 13:55:13.112511 23993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.129:35903 every 8 connection(s)
I20240501 13:55:13.120513 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:13.137717 23998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:13.138183 23999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:13.152310 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:13.154213 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:13.155282 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:13.155582 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:13.155800 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571713155780 us; error 0 us; skew 500 ppm
I20240501 13:55:13.156687 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:13.162832 23834 webserver.cc:460] Webserver started at http://127.23.70.130:38649/ using document root <none> and password file <none>
I20240501 13:55:13.163591 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:13.163859 23994 heartbeater.cc:347] Connected to a master server at 127.23.70.190:42367
I20240501 13:55:13.163879 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:13.164299 23994 heartbeater.cc:464] Registering TS with master...
I20240501 13:55:13.164332 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:13.165277 23994 heartbeater.cc:511] Master 127.23.70.190:42367 requested a full tablet report, sending...
I20240501 13:55:13.165966 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-1-root/instance:
uuid: "a22335d124c946bda7b03079b5661572"
format_stamp: "Formatted at 2024-05-01 13:55:13 on dist-test-slave-s72h"
I20240501 13:55:13.167968 23866 ts_manager.cc:194] Registered new tserver with Master: 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903)
I20240501 13:55:13.170177 23866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57956
I20240501 13:55:13.173884 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20240501 13:55:13.179667 24006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.180722 23834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20240501 13:55:13.181031 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
uuid: "a22335d124c946bda7b03079b5661572"
format_stamp: "Formatted at 2024-05-01 13:55:13 on dist-test-slave-s72h"
I20240501 13:55:13.181313 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:13.213567 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:13.214876 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:13.216226 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:13.220572 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:13.220788 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.221021 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:13.221175 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.334692 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.130:36467
I20240501 13:55:13.334872 24068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.130:36467 every 8 connection(s)
I20240501 13:55:13.344434 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:13.364754 24072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:13.366277 24069 heartbeater.cc:347] Connected to a master server at 127.23.70.190:42367
I20240501 13:55:13.366741 24069 heartbeater.cc:464] Registering TS with master...
W20240501 13:55:13.367414 24073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:13.367635 24069 heartbeater.cc:511] Master 127.23.70.190:42367 requested a full tablet report, sending...
W20240501 13:55:13.368722 24075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:13.370044 23866 ts_manager.cc:194] Registered new tserver with Master: a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:13.370762 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:13.371706 23866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57964
I20240501 13:55:13.371912 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:13.372186 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:13.372424 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571713372406 us; error 0 us; skew 500 ppm
I20240501 13:55:13.373569 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:13.376603 23834 webserver.cc:460] Webserver started at http://127.23.70.131:44823/ using document root <none> and password file <none>
I20240501 13:55:13.377240 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:13.377492 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:13.377820 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:13.379245 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-2-root/instance:
uuid: "06a90d5210104a7b802c94cbe78f375d"
format_stamp: "Formatted at 2024-05-01 13:55:13 on dist-test-slave-s72h"
I20240501 13:55:13.385447 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20240501 13:55:13.389899 24080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.390815 23834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 13:55:13.391104 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
uuid: "06a90d5210104a7b802c94cbe78f375d"
format_stamp: "Formatted at 2024-05-01 13:55:13 on dist-test-slave-s72h"
I20240501 13:55:13.391376 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714571711146223-23834-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:13.412043 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:13.413312 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:13.415459 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:13.419009 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:13.419246 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.419493 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:13.419651 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:13.534056 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.131:35875
I20240501 13:55:13.534305 24142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.131:35875 every 8 connection(s)
I20240501 13:55:13.554823 24143 heartbeater.cc:347] Connected to a master server at 127.23.70.190:42367
I20240501 13:55:13.555303 24143 heartbeater.cc:464] Registering TS with master...
I20240501 13:55:13.556411 24143 heartbeater.cc:511] Master 127.23.70.190:42367 requested a full tablet report, sending...
I20240501 13:55:13.558707 23866 ts_manager.cc:194] Registered new tserver with Master: 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:13.559253 23834 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01922807s
I20240501 13:55:13.560559 23866 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57978
I20240501 13:55:13.569276 23834 test_util.cc:268] Using random seed: -705830436
I20240501 13:55:13.626371 23866 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57990:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 13:55:13.631927 23866 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:55:13.768754 24107 tablet_service.cc:1467] Processing CreateTablet for tablet 5e54c75c9d07413a9df84bea534f69f2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 13:55:13.769366 24106 tablet_service.cc:1467] Processing CreateTablet for tablet 122b6f8fb7ab4b89bf8751df210ceaaa (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
W20240501 13:55:13.770771 23914 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:55:13.768801 24108 tablet_service.cc:1467] Processing CreateTablet for tablet d7286b7bc0974727ae5650de0f78a3b3 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 13:55:13.771625 24105 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c504bf8264d589bce4c3c4d29b477 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 13:55:13.777966 24034 tablet_service.cc:1467] Processing CreateTablet for tablet d7286b7bc0974727ae5650de0f78a3b3 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 13:55:13.779206 23959 tablet_service.cc:1467] Processing CreateTablet for tablet d7286b7bc0974727ae5650de0f78a3b3 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 13:55:13.785398 24107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e54c75c9d07413a9df84bea534f69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.784914 24104 tablet_service.cc:1467] Processing CreateTablet for tablet bdb52db7fa1340c68d6845e53b1b5e49 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 13:55:13.793972 24034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7286b7bc0974727ae5650de0f78a3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.796164 23959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7286b7bc0974727ae5650de0f78a3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.799356 24106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b6f8fb7ab4b89bf8751df210ceaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.810096 24108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7286b7bc0974727ae5650de0f78a3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.801045 24032 tablet_service.cc:1467] Processing CreateTablet for tablet 122b6f8fb7ab4b89bf8751df210ceaaa (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240501 13:55:13.799337 24103 tablet_service.cc:1467] Processing CreateTablet for tablet 836a46d915f6442eb009a671446afc63 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 13:55:13.808852 23958 tablet_service.cc:1467] Processing CreateTablet for tablet 5e54c75c9d07413a9df84bea534f69f2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 13:55:13.821259 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b6f8fb7ab4b89bf8751df210ceaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.821316 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 5e54c75c9d07413a9df84bea534f69f2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 13:55:13.811532 23955 tablet_service.cc:1467] Processing CreateTablet for tablet bdb52db7fa1340c68d6845e53b1b5e49 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 13:55:13.814908 24030 tablet_service.cc:1467] Processing CreateTablet for tablet bdb52db7fa1340c68d6845e53b1b5e49 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 13:55:13.825906 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c504bf8264d589bce4c3c4d29b477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.828308 23953 tablet_service.cc:1467] Processing CreateTablet for tablet f59ff8d5e98045a093914244a83d1ce4 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 13:55:13.825124 24028 tablet_service.cc:1467] Processing CreateTablet for tablet f59ff8d5e98045a093914244a83d1ce4 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 13:55:13.825032 23956 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c504bf8264d589bce4c3c4d29b477 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 13:55:13.832373 24102 tablet_service.cc:1467] Processing CreateTablet for tablet f59ff8d5e98045a093914244a83d1ce4 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 13:55:13.834353 23957 tablet_service.cc:1467] Processing CreateTablet for tablet 122b6f8fb7ab4b89bf8751df210ceaaa (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240501 13:55:13.824270 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 836a46d915f6442eb009a671446afc63 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 13:55:13.831063 23952 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac44a39a04d458badaf6ced34c24bc2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 13:55:13.826512 24031 tablet_service.cc:1467] Processing CreateTablet for tablet bd2c504bf8264d589bce4c3c4d29b477 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 13:55:13.846414 23954 tablet_service.cc:1467] Processing CreateTablet for tablet 836a46d915f6442eb009a671446afc63 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 13:55:13.850029 24103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836a46d915f6442eb009a671446afc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.850836 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb52db7fa1340c68d6845e53b1b5e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.856472 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e54c75c9d07413a9df84bea534f69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.860363 23954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836a46d915f6442eb009a671446afc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.861752 24102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59ff8d5e98045a093914244a83d1ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.883919 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59ff8d5e98045a093914244a83d1ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.854640 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac44a39a04d458badaf6ced34c24bc2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 13:55:13.852324 24101 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac44a39a04d458badaf6ced34c24bc2 (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 13:55:13.887161 24104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb52db7fa1340c68d6845e53b1b5e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.892428 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac44a39a04d458badaf6ced34c24bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.894673 24031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c504bf8264d589bce4c3c4d29b477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.897583 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836a46d915f6442eb009a671446afc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.899305 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f59ff8d5e98045a093914244a83d1ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.884052 24026 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb8a2eeba14b759fb7591a5793a99d (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 13:55:13.903321 23951 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb8a2eeba14b759fb7591a5793a99d (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 13:55:13.900373 24100 tablet_service.cc:1467] Processing CreateTablet for tablet 48cb8a2eeba14b759fb7591a5793a99d (DEFAULT_TABLE table=test-workload [id=e594be1c063c4b21afacdb08cdf19505]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 13:55:13.907891 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac44a39a04d458badaf6ced34c24bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.917244 24100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cb8a2eeba14b759fb7591a5793a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.925740 23951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cb8a2eeba14b759fb7591a5793a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.924532 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdb52db7fa1340c68d6845e53b1b5e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.929035 23958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e54c75c9d07413a9df84bea534f69f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.954088 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac44a39a04d458badaf6ced34c24bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.965297 24026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48cb8a2eeba14b759fb7591a5793a99d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.975996 23957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122b6f8fb7ab4b89bf8751df210ceaaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:13.987210 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2c504bf8264d589bce4c3c4d29b477. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:14.153152 24161 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.172181 24162 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.175271 23994 heartbeater.cc:503] Master 127.23.70.190:42367 was elected leader, sending a full tablet report...
I20240501 13:55:14.193444 24163 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.211598 24161 tablet_bootstrap.cc:654] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.233551 24162 tablet_bootstrap.cc:654] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.234155 24163 tablet_bootstrap.cc:654] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.257062 24161 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.258088 24161 ts_tablet_manager.cc:1367] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.161s user 0.021s sys 0.012s
I20240501 13:55:14.263062 24161 raft_consensus.cc:348] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.264081 24161 raft_consensus.cc:374] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.264492 24161 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.264619 24162 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.265242 24162 ts_tablet_manager.cc:1367] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.096s user 0.016s sys 0.006s
I20240501 13:55:14.265508 24161 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.268416 24162 raft_consensus.cc:348] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.269438 24162 raft_consensus.cc:374] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.269861 24162 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.270903 24162 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.274206 24069 heartbeater.cc:503] Master 127.23.70.190:42367 was elected leader, sending a full tablet report...
I20240501 13:55:14.280184 24161 ts_tablet_manager.cc:1398] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.022s user 0.002s sys 0.012s
I20240501 13:55:14.281356 24161 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.286106 24163 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.286576 24163 ts_tablet_manager.cc:1367] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.114s user 0.026s sys 0.026s
I20240501 13:55:14.288414 24161 tablet_bootstrap.cc:654] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.288570 24162 ts_tablet_manager.cc:1398] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.023s user 0.005s sys 0.007s
I20240501 13:55:14.295398 24163 raft_consensus.cc:348] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.296221 24163 raft_consensus.cc:374] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.296543 24163 raft_consensus.cc:729] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.297232 24163 consensus_queue.cc:260] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.300410 24143 heartbeater.cc:503] Master 127.23.70.190:42367 was elected leader, sending a full tablet report...
I20240501 13:55:14.301116 24161 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.301680 24161 ts_tablet_manager.cc:1367] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
W20240501 13:55:14.303125 24144 tablet.cc:2369] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:55:14.304198 24162 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.304471 24161 raft_consensus.cc:348] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.305214 24161 raft_consensus.cc:374] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.305516 24161 raft_consensus.cc:729] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.306219 24161 consensus_queue.cc:260] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.309988 24161 ts_tablet_manager.cc:1398] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20240501 13:55:14.311764 24161 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.318105 24163 ts_tablet_manager.cc:1398] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20240501 13:55:14.318599 24161 tablet_bootstrap.cc:654] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.321070 24169 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.325052 24169 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.331985 24162 tablet_bootstrap.cc:654] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.324637 24163 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.349675 24169 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:14.353987 24161 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.354562 24162 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.354629 24161 ts_tablet_manager.cc:1367] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.012s
I20240501 13:55:14.356109 24162 ts_tablet_manager.cc:1367] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.007s
I20240501 13:55:14.362304 24161 raft_consensus.cc:348] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.363127 24161 raft_consensus.cc:374] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.363536 24161 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.363312 24163 tablet_bootstrap.cc:654] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.365583 24162 raft_consensus.cc:348] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.366360 24162 raft_consensus.cc:374] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.366658 24162 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.367347 24162 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.369585 24162 ts_tablet_manager.cc:1398] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 13:55:14.364390 24161 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.374611 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:14.375962 24161 ts_tablet_manager.cc:1398] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 13:55:14.376999 24161 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
W20240501 13:55:14.379856 24081 leader_election.cc:341] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a22335d124c946bda7b03079b5661572 (127.23.70.130:36467): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.382444 24162 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.384507 24161 tablet_bootstrap.cc:654] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.391263 24162 tablet_bootstrap.cc:654] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.401515 24167 raft_consensus.cc:482] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.401775 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.402166 24167 raft_consensus.cc:504] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
W20240501 13:55:14.405880 24082 leader_election.cc:341] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.406553 24082 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d; no voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572
I20240501 13:55:14.408833 24163 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.407967 24167 leader_election.cc:288] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.409168 24169 raft_consensus.cc:2726] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:14.411640 24163 ts_tablet_manager.cc:1367] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.002s
I20240501 13:55:14.421828 24163 raft_consensus.cc:348] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.427974 24163 raft_consensus.cc:374] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.433058 24163 raft_consensus.cc:729] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.433940 24163 consensus_queue.cc:260] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.437273 24163 ts_tablet_manager.cc:1398] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.025s user 0.003s sys 0.006s
I20240501 13:55:14.438218 24163 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.441833 24161 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.443030 24161 ts_tablet_manager.cc:1367] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.023s
I20240501 13:55:14.452713 24161 raft_consensus.cc:348] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.454037 24162 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.454524 24161 raft_consensus.cc:374] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.455487 24162 ts_tablet_manager.cc:1367] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.016s
I20240501 13:55:14.455657 24161 raft_consensus.cc:729] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.458729 24161 consensus_queue.cc:260] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.470455 24161 ts_tablet_manager.cc:1398] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.026s user 0.009s sys 0.001s
I20240501 13:55:14.472481 24161 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.473158 24163 tablet_bootstrap.cc:654] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.470779 24162 raft_consensus.cc:348] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.474390 24162 raft_consensus.cc:374] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.475189 24162 raft_consensus.cc:729] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.477094 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.478190 24162 consensus_queue.cc:260] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
W20240501 13:55:14.481642 24008 leader_election.cc:341] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.484855 24162 ts_tablet_manager.cc:1398] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.028s user 0.005s sys 0.004s
I20240501 13:55:14.486516 24162 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.498617 24163 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.499258 24163 ts_tablet_manager.cc:1367] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.011s
I20240501 13:55:14.499150 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.500062 24118 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.502147 24010 leader_election.cc:302] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters: 093b61249a29465596a6542c3b6da169
I20240501 13:55:14.503247 24167 raft_consensus.cc:2781] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.503707 24167 raft_consensus.cc:482] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.504395 24167 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.504534 24163 raft_consensus.cc:348] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.505947 24163 raft_consensus.cc:374] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.506390 24163 raft_consensus.cc:729] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.507608 24161 tablet_bootstrap.cc:654] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.507478 24163 consensus_queue.cc:260] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.510303 24163 ts_tablet_manager.cc:1398] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240501 13:55:14.514400 24162 tablet_bootstrap.cc:654] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.515239 24167 raft_consensus.cc:504] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.518338 24167 leader_election.cc:288] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.518640 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.519367 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.519943 24118 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:55:14.520112 24008 leader_election.cc:341] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.522548 24163 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.522284 24167 raft_consensus.cc:482] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.525087 24167 raft_consensus.cc:504] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.528051 24167 leader_election.cc:288] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.528090 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.529098 24118 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:14.528998 23969 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.529968 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.530419 24010 leader_election.cc:302] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters: 093b61249a29465596a6542c3b6da169
I20240501 13:55:14.531401 24163 tablet_bootstrap.cc:654] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
W20240501 13:55:14.531854 24010 leader_election.cc:341] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.533273 24008 leader_election.cc:302] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:14.540434 24167 raft_consensus.cc:2781] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:14.540645 24161 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.548830 24161 ts_tablet_manager.cc:1367] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.013s
I20240501 13:55:14.553550 24161 raft_consensus.cc:348] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.554672 24161 raft_consensus.cc:374] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.555070 24161 raft_consensus.cc:729] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.558073 24188 raft_consensus.cc:2781] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.558781 24188 raft_consensus.cc:482] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.559376 24188 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.558816 24161 consensus_queue.cc:260] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.560089 24163 tablet_bootstrap.cc:492] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.560650 24163 ts_tablet_manager.cc:1367] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.021s
I20240501 13:55:14.563741 24162 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.564425 24162 ts_tablet_manager.cc:1367] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.078s user 0.013s sys 0.027s
I20240501 13:55:14.567034 24167 raft_consensus.cc:686] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:14.568013 24167 consensus_queue.cc:237] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.568173 24162 raft_consensus.cc:348] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.568895 24161 ts_tablet_manager.cc:1398] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.019s user 0.010s sys 0.005s
I20240501 13:55:14.569149 24162 raft_consensus.cc:374] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.569464 24162 raft_consensus.cc:729] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.569995 24161 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.570307 24162 consensus_queue.cc:260] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.571081 24163 raft_consensus.cc:348] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.571870 24163 raft_consensus.cc:374] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.572660 24162 ts_tablet_manager.cc:1398] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240501 13:55:14.573637 24162 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.574107 24188 raft_consensus.cc:504] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.576764 24188 leader_election.cc:288] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.572163 24163 raft_consensus.cc:729] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.577967 24163 consensus_queue.cc:260] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.578848 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.579428 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.580005 23969 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:55:14.580614 24010 leader_election.cc:341] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.583550 24163 ts_tablet_manager.cc:1398] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.021s user 0.005s sys 0.003s
I20240501 13:55:14.584565 24163 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.593875 24171 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.594383 24171 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.592875 24190 raft_consensus.cc:482] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.596894 23969 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:14.599830 24171 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.597277 24190 raft_consensus.cc:504] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.604403 24162 tablet_bootstrap.cc:654] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.608923 24163 tablet_bootstrap.cc:654] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.613193 24169 raft_consensus.cc:482] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.614678 24169 raft_consensus.cc:504] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.616815 24169 leader_election.cc:288] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:14.618062 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.618630 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
W20240501 13:55:14.620442 24082 leader_election.cc:341] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:55:14.621387 24081 leader_election.cc:341] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a22335d124c946bda7b03079b5661572 (127.23.70.130:36467): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.620700 23865 catalog_manager.cc:5484] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:14.625838 24081 leader_election.cc:302] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d; no voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572
I20240501 13:55:14.626938 24161 tablet_bootstrap.cc:654] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.629438 24169 raft_consensus.cc:2726] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:14.632556 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.633296 24117 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.634915 24010 leader_election.cc:302] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.627699 24190 leader_election.cc:288] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.636265 24167 raft_consensus.cc:2781] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.661211 24190 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.661243 24167 raft_consensus.cc:482] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.663692 24190 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.653204 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.668632 24190 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.664266 24167 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.670950 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.648473 24008 leader_election.cc:302] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:14.672875 23969 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
W20240501 13:55:14.677294 24010 leader_election.cc:341] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.679078 24163 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.679482 24188 raft_consensus.cc:2781] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:14.679659 24163 ts_tablet_manager.cc:1367] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.020s
I20240501 13:55:14.679965 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.680049 24188 raft_consensus.cc:686] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:14.680506 24189 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.680984 24189 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.680909 23969 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.681306 24188 consensus_queue.cc:237] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.684918 24162 tablet_bootstrap.cc:492] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.685437 24162 ts_tablet_manager.cc:1367] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.112s user 0.019s sys 0.008s
I20240501 13:55:14.695623 24189 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875), 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903)
I20240501 13:55:14.698711 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.698782 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.700558 24167 raft_consensus.cc:504] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
W20240501 13:55:14.700963 24010 leader_election.cc:341] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:55:14.704331 23935 leader_election.cc:341] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.705677 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:14.706336 24044 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 0.
I20240501 13:55:14.705823 24162 raft_consensus.cc:348] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.684959 24008 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:14.709527 23932 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:14.709187 24167 leader_election.cc:288] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.708680 24162 raft_consensus.cc:374] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.713166 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.713482 24171 raft_consensus.cc:2781] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.713384 24161 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.714048 24118 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.714702 24161 ts_tablet_manager.cc:1367] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.145s user 0.020s sys 0.015s
I20240501 13:55:14.715984 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.714354 24171 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.717142 24171 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.717451 24161 raft_consensus.cc:348] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.718194 24161 raft_consensus.cc:374] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.718152 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.718659 24161 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.718904 23969 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.713198 24162 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.719897 24162 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.721199 24163 raft_consensus.cc:348] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.722041 24163 raft_consensus.cc:374] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.722317 24162 ts_tablet_manager.cc:1398] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.037s user 0.010s sys 0.000s
W20240501 13:55:14.723403 24008 leader_election.cc:341] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.723858 24008 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169
I20240501 13:55:14.725013 24118 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:14.725857 24167 raft_consensus.cc:2781] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.722322 24163 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.726868 24167 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.727264 24167 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.727846 24189 raft_consensus.cc:2726] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:14.727521 24163 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.730041 24010 leader_election.cc:302] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.731550 24189 raft_consensus.cc:2781] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:14.732012 24189 raft_consensus.cc:686] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:14.733122 24189 consensus_queue.cc:237] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.733743 24162 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.736400 24167 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.735545 24171 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.742933 23969 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:14.743213 24163 ts_tablet_manager.cc:1398] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.063s user 0.006s sys 0.010s
I20240501 13:55:14.730754 24161 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.747292 24171 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.748200 24167 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.749569 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.749894 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:14.756682 24163 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
W20240501 13:55:14.759541 24010 leader_election.cc:341] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.761147 24044 raft_consensus.cc:2379] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 093b61249a29465596a6542c3b6da169 in current term 1: Already voted for candidate a22335d124c946bda7b03079b5661572 in this term.
I20240501 13:55:14.750080 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.761353 24161 ts_tablet_manager.cc:1398] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.046s user 0.006s sys 0.002s
I20240501 13:55:14.762380 24161 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
W20240501 13:55:14.764266 23935 leader_election.cc:341] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.765925 23932 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 093b61249a29465596a6542c3b6da169; no voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572
I20240501 13:55:14.769259 24163 tablet_bootstrap.cc:654] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.772715 24171 raft_consensus.cc:2726] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240501 13:55:14.780792 23914 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:55:14.763149 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.788491 23969 raft_consensus.cc:2379] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a22335d124c946bda7b03079b5661572 in current term 1: Already voted for candidate 093b61249a29465596a6542c3b6da169 in this term.
I20240501 13:55:14.789619 24008 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a22335d124c946bda7b03079b5661572; no voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169
I20240501 13:55:14.790274 24189 raft_consensus.cc:2726] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:55:14.790850 24161 tablet_bootstrap.cc:654] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.792829 24162 tablet_bootstrap.cc:654] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.798926 24163 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.806370 24163 ts_tablet_manager.cc:1367] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.005s
I20240501 13:55:14.810653 24161 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.812296 24163 raft_consensus.cc:348] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.813490 24163 raft_consensus.cc:374] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.813910 24163 raft_consensus.cc:729] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.815263 24161 ts_tablet_manager.cc:1367] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.020s
I20240501 13:55:14.815274 24163 consensus_queue.cc:260] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.819046 24161 raft_consensus.cc:348] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.822260 24161 raft_consensus.cc:374] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.821818 24162 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.823045 24162 ts_tablet_manager.cc:1367] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.090s user 0.012s sys 0.012s
I20240501 13:55:14.819626 24163 ts_tablet_manager.cc:1398] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20240501 13:55:14.824995 24163 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.825771 24162 raft_consensus.cc:348] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.826567 24161 raft_consensus.cc:729] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.828876 24161 consensus_queue.cc:260] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.826601 24162 raft_consensus.cc:374] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.832165 24162 raft_consensus.cc:729] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.832888 24171 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.836226 24171 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.833070 24162 consensus_queue.cc:260] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.837373 24163 tablet_bootstrap.cc:654] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.839249 24162 ts_tablet_manager.cc:1398] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240501 13:55:14.834334 24161 ts_tablet_manager.cc:1398] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.018s user 0.001s sys 0.006s
I20240501 13:55:14.840077 24171 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:14.841029 24161 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.832720 23865 catalog_manager.cc:5484] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:14.842403 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:14.842670 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.843080 24044 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 0.
I20240501 13:55:14.843232 24162 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.843292 24117 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 0.
I20240501 13:55:14.844899 23932 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.843942 23865 catalog_manager.cc:5484] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:14.846832 24171 raft_consensus.cc:2781] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.861001 24171 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.861408 24171 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.851678 24161 tablet_bootstrap.cc:654] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.857193 24162 tablet_bootstrap.cc:654] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.860106 24163 tablet_bootstrap.cc:492] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.862934 24163 ts_tablet_manager.cc:1367] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.011s
I20240501 13:55:14.866413 24163 raft_consensus.cc:348] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.867177 24163 raft_consensus.cc:374] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.868515 24163 raft_consensus.cc:729] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.869242 24163 consensus_queue.cc:260] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.873476 24200 raft_consensus.cc:482] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.874161 24200 raft_consensus.cc:504] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.872092 24171 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.878484 24200 leader_election.cc:288] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.879228 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:14.880605 24171 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Requested vote from peers 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:14.881572 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
W20240501 13:55:14.884037 23932 leader_election.cc:341] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a22335d124c946bda7b03079b5661572 (127.23.70.130:36467): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.884274 24161 tablet_bootstrap.cc:492] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.884814 24161 ts_tablet_manager.cc:1367] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.044s user 0.015s sys 0.009s
W20240501 13:55:14.885324 23935 leader_election.cc:341] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.885759 23935 leader_election.cc:302] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 093b61249a29465596a6542c3b6da169; no voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572
I20240501 13:55:14.887189 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:14.887831 24044 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.888067 24161 raft_consensus.cc:348] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.888792 24161 raft_consensus.cc:374] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.889045 24161 raft_consensus.cc:729] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.892119 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.892767 24117 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.896508 24161 consensus_queue.cc:260] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.897575 24171 raft_consensus.cc:2726] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:14.907469 24189 raft_consensus.cc:482] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.900281 24163 ts_tablet_manager.cc:1398] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.037s user 0.007s sys 0.003s
I20240501 13:55:14.902060 24161 ts_tablet_manager.cc:1398] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240501 13:55:14.907981 24189 raft_consensus.cc:504] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.908773 24163 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:14.910115 24189 leader_election.cc:288] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.911619 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:14.912279 23969 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.913264 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.913915 24118 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.915208 24010 leader_election.cc:302] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.915973 24167 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:14.902746 24044 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 1.
I20240501 13:55:14.907289 24117 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 1.
I20240501 13:55:14.916317 24163 tablet_bootstrap.cc:654] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.916975 24189 raft_consensus.cc:2781] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.918661 24189 raft_consensus.cc:482] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.919024 24189 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.919924 24161 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: Bootstrap starting.
I20240501 13:55:14.916456 24167 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.926523 24167 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.917275 23932 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.927945 24171 raft_consensus.cc:2781] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:14.928455 24171 raft_consensus.cc:686] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:14.929282 24171 consensus_queue.cc:237] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:14.933576 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:14.934278 24117 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:14.935596 24010 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:14.938014 24161 tablet_bootstrap.cc:654] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:14.938431 24167 raft_consensus.cc:2781] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:14.938845 24167 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:14.939199 24167 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.944818 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
W20240501 13:55:14.946389 24008 leader_election.cc:341] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:14.957767 24162 tablet_bootstrap.cc:492] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:14.958163 24162 ts_tablet_manager.cc:1367] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.115s user 0.020s sys 0.002s
I20240501 13:55:14.959455 24161 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: No bootstrap required, opened a new log
I20240501 13:55:14.960181 24161 ts_tablet_manager.cc:1367] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.014s
I20240501 13:55:14.965647 24161 raft_consensus.cc:348] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.966776 24163 tablet_bootstrap.cc:492] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:14.967080 24161 raft_consensus.cc:374] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.967298 24163 ts_tablet_manager.cc:1367] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.030s
I20240501 13:55:14.965456 24167 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.969584 24167 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.960297 24162 raft_consensus.cc:348] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.970582 24162 raft_consensus.cc:374] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.970896 24162 raft_consensus.cc:729] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.970294 23865 catalog_manager.cc:5484] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:14.971796 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.956602 24189 raft_consensus.cc:504] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.967456 24161 raft_consensus.cc:729] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.976454 24189 leader_election.cc:288] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:14.975006 24162 consensus_queue.cc:260] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.978578 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.977057 24161 consensus_queue.cc:260] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.979305 24118 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.979713 24162 ts_tablet_manager.cc:1398] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20240501 13:55:14.980852 24162 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:14.985437 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:14.985239 24163 raft_consensus.cc:348] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.986044 24163 raft_consensus.cc:374] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:14.972968 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:14.986343 24163 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:14.986595 24117 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.987114 24161 ts_tablet_manager.cc:1398] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
I20240501 13:55:14.987025 24163 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:14.988075 23969 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:14.995137 23969 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.005832 24163 ts_tablet_manager.cc:1398] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.038s user 0.006s sys 0.001s
I20240501 13:55:14.986065 23968 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.011852 24162 tablet_bootstrap.cc:654] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:15.012130 24118 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.012655 24189 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:55:15.014225 24163 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: Bootstrap starting.
I20240501 13:55:15.020738 23968 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.021132 24008 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:15.022156 24010 leader_election.cc:302] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:15.023515 24167 raft_consensus.cc:2781] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:15.024058 24167 raft_consensus.cc:686] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:15.024214 24188 raft_consensus.cc:2781] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:15.024770 24188 raft_consensus.cc:686] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:15.025880 24188 consensus_queue.cc:237] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.025048 24167 consensus_queue.cc:237] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.044739 24163 tablet_bootstrap.cc:654] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:15.048712 24117 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.059221 23865 catalog_manager.cc:5484] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:15.055739 24163 tablet_bootstrap.cc:492] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: No bootstrap required, opened a new log
I20240501 13:55:15.062408 24163 ts_tablet_manager.cc:1367] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.001s
I20240501 13:55:15.062800 23865 catalog_manager.cc:5484] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:15.065937 24163 raft_consensus.cc:348] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.066753 24163 raft_consensus.cc:374] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:15.067024 24163 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Initialized, Role: FOLLOWER
I20240501 13:55:15.067669 24163 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.073884 24163 ts_tablet_manager.cc:1398] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240501 13:55:15.090981 24162 tablet_bootstrap.cc:492] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:15.091626 24162 ts_tablet_manager.cc:1367] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.111s user 0.014s sys 0.013s
I20240501 13:55:15.095758 24162 raft_consensus.cc:348] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.096693 24162 raft_consensus.cc:374] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:15.097043 24162 raft_consensus.cc:729] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:15.097715 24162 consensus_queue.cc:260] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.099992 24162 ts_tablet_manager.cc:1398] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 13:55:15.100873 24162 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: Bootstrap starting.
I20240501 13:55:15.107225 24162 tablet_bootstrap.cc:654] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:15.112000 23968 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.112994 24162 tablet_bootstrap.cc:492] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: No bootstrap required, opened a new log
I20240501 13:55:15.113544 24162 ts_tablet_manager.cc:1367] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20240501 13:55:15.115988 24162 raft_consensus.cc:348] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.116705 24162 raft_consensus.cc:374] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:15.116995 24162 raft_consensus.cc:729] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Initialized, Role: FOLLOWER
I20240501 13:55:15.117635 24162 consensus_queue.cc:260] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.119354 24162 ts_tablet_manager.cc:1398] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 13:55:15.120929 24188 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:55:15.133852 24188 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.136629 24171 raft_consensus.cc:482] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:15.138347 24171 raft_consensus.cc:504] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.142386 24171 leader_election.cc:288] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:15.142542 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:15.143194 24044 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 0.
I20240501 13:55:15.144232 23932 leader_election.cc:302] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:15.145434 24188 consensus_queue.cc:1035] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.146001 24117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:15.146641 24117 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 0.
I20240501 13:55:15.148013 24171 raft_consensus.cc:2781] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:15.148473 24171 raft_consensus.cc:482] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:15.148886 24171 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.150048 24169 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:15.150681 24169 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.153447 24169 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:15.155371 23966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:15.156941 23966 raft_consensus.cc:2379] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06a90d5210104a7b802c94cbe78f375d in current term 1: Already voted for candidate 093b61249a29465596a6542c3b6da169 in this term.
I20240501 13:55:15.156735 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572" is_pre_election: true
I20240501 13:55:15.157846 24044 raft_consensus.cc:2379] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06a90d5210104a7b802c94cbe78f375d in current term 1: Already voted for candidate a22335d124c946bda7b03079b5661572 in this term.
I20240501 13:55:15.160013 24081 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d; no voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572
I20240501 13:55:15.162423 24169 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.178635 24117 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.178781 24190 consensus_queue.cc:1035] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.187788 24171 raft_consensus.cc:504] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.176162 24169 raft_consensus.cc:2726] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:15.192278 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:15.192976 24044 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.193289 24116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:15.193897 24116 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.194447 24188 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.198480 24171 leader_election.cc:288] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:15.204025 24116 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 1.
I20240501 13:55:15.205612 23935 leader_election.cc:302] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:15.206851 24171 raft_consensus.cc:2781] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:15.207780 24171 raft_consensus.cc:686] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:15.208038 24044 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 1.
I20240501 13:55:15.210644 24171 consensus_queue.cc:237] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.238353 23865 catalog_manager.cc:5484] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 0 to 1, leader changed from <none> to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:15.403744 24171 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.415233 24171 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:55:15.454679 24167 raft_consensus.cc:482] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:15.455170 24167 raft_consensus.cc:504] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.456902 24167 leader_election.cc:288] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:15.458015 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:15.457774 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:15.458743 24118 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:15.458886 23967 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 0.
I20240501 13:55:15.459837 24010 leader_election.cc:302] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:15.460708 24167 raft_consensus.cc:2781] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:15.461014 24167 raft_consensus.cc:482] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:15.461264 24167 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.466075 24167 raft_consensus.cc:504] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.468178 24167 leader_election.cc:288] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:15.468545 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:15.468860 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:15.469221 23967 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.469350 24118 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:15.474993 24188 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.476114 24118 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.477067 24010 leader_election.cc:302] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:15.478615 24188 raft_consensus.cc:2781] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:15.479267 24188 raft_consensus.cc:686] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:15.479348 23967 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:15.481411 24188 consensus_queue.cc:237] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:15.491638 24190 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.505301 24189 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.525151 23864 catalog_manager.cc:5484] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:15.536723 24167 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.602519 24231 consensus_queue.cc:1035] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.619989 24171 consensus_queue.cc:1035] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:55:15.788230 23914 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:55:15.961469 24167 consensus_queue.cc:1035] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:15.963330 24167 consensus_queue.cc:1035] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:16.322173 24190 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:16.322695 24190 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.324254 24190 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:16.325209 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169" is_pre_election: true
I20240501 13:55:16.325223 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d" is_pre_election: true
I20240501 13:55:16.325804 23968 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:16.325875 24118 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 1.
I20240501 13:55:16.326795 24008 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:16.327448 24190 raft_consensus.cc:2781] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:55:16.327798 24190 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:16.328027 24190 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.332291 24190 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.333801 24190 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:16.334663 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:16.335192 23968 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.335088 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:16.335666 24118 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.339466 23968 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 2.
I20240501 13:55:16.339823 24118 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 2.
I20240501 13:55:16.340556 24008 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:16.341357 24190 raft_consensus.cc:2781] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:16.341804 24190 raft_consensus.cc:686] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:16.342475 24190 consensus_queue.cc:237] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.349378 23865 catalog_manager.cc:5484] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 0 to 2, leader changed from <none> to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 2 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:16.560083 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:57452
I20240501 13:55:16.560741 24044 raft_consensus.cc:595] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:16.565995 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:57452
I20240501 13:55:16.566610 24044 raft_consensus.cc:595] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:16.571314 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:57452
I20240501 13:55:16.571857 24044 raft_consensus.cc:595] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:16.585717 23968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49"
dest_uuid: "093b61249a29465596a6542c3b6da169"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:49292
I20240501 13:55:16.586410 23968 raft_consensus.cc:595] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:16.599922 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:57452
I20240501 13:55:16.600482 24044 raft_consensus.cc:595] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:16.710773 24189 raft_consensus.cc:979] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:16.711196 24190 raft_consensus.cc:1067] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:16.712678 23968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:16.713187 23968 raft_consensus.cc:482] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:16.713441 23968 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.717749 23968 raft_consensus.cc:504] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.719683 23968 leader_election.cc:288] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:16.720613 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:16.720731 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:16.721330 24044 raft_consensus.cc:3032] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:16.721411 24118 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.721805 24044 raft_consensus.cc:729] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:16.722615 24044 consensus_queue.cc:260] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.723960 24044 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.728339 24118 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:16.729178 24115 raft_consensus.cc:1226] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Rejecting Update request from peer a22335d124c946bda7b03079b5661572 for earlier term 1. Current term is 2. Ops: []
I20240501 13:55:16.729635 23935 leader_election.cc:302] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:16.730495 24239 raft_consensus.cc:2781] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:16.730573 24044 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:16.732100 24239 raft_consensus.cc:686] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:16.732932 24239 consensus_queue.cc:237] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.739861 23865 catalog_manager.cc:5484] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:16.798862 24115 raft_consensus.cc:1261] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 13:55:16.800242 24190 consensus_queue.cc:1035] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:16.811245 23968 raft_consensus.cc:1261] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 13:55:16.813035 24190 consensus_queue.cc:1035] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:16.876314 24189 raft_consensus.cc:979] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: : Instructing follower 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:16.876785 24189 raft_consensus.cc:1067] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:16.878119 24115 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:55502
I20240501 13:55:16.878620 24115 raft_consensus.cc:482] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:16.878890 24115 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.883795 24115 raft_consensus.cc:504] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.885349 24115 leader_election.cc:288] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:16.886289 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:16.886370 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cb8a2eeba14b759fb7591a5793a99d" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:16.886945 23968 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.887034 24044 raft_consensus.cc:3032] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:16.887357 24044 raft_consensus.cc:729] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:16.887962 24044 consensus_queue.cc:260] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.889340 24044 raft_consensus.cc:3037] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.891701 23968 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:16.892834 24082 leader_election.cc:302] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:16.893743 24252 raft_consensus.cc:2781] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:16.895020 24252 raft_consensus.cc:686] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Becoming Leader. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:16.895112 24044 raft_consensus.cc:2454] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:16.895882 24252 consensus_queue.cc:237] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.902360 23864 catalog_manager.cc:5484] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 06a90d5210104a7b802c94cbe78f375d (127.23.70.131). New cstate: current_term: 2 leader_uuid: "06a90d5210104a7b802c94cbe78f375d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:16.908190 24190 raft_consensus.cc:979] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:16.908634 24189 raft_consensus.cc:1067] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:16.910187 23968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:16.910691 23968 raft_consensus.cc:482] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:16.910974 23968 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.915374 23968 raft_consensus.cc:504] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.916880 23968 leader_election.cc:288] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:16.917786 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:16.917904 24115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:16.918306 24044 raft_consensus.cc:3032] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:16.918407 24115 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.918608 24044 raft_consensus.cc:729] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:16.919358 24044 consensus_queue.cc:260] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.920624 24044 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:16.923451 24115 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:16.924496 23935 leader_election.cc:302] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:16.925271 24044 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:16.925346 24239 raft_consensus.cc:2781] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:16.925887 24239 raft_consensus.cc:686] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:16.926717 24239 consensus_queue.cc:237] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:16.933194 23864 catalog_manager.cc:5484] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:17.012907 24250 raft_consensus.cc:979] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:17.013341 24239 raft_consensus.cc:1067] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:17.014621 24044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57440
I20240501 13:55:17.015091 24044 raft_consensus.cc:482] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:17.015362 24044 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.019742 24044 raft_consensus.cc:504] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.021276 24044 leader_election.cc:288] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:17.022192 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:17.022253 24115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdb52db7fa1340c68d6845e53b1b5e49" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:17.022778 23968 raft_consensus.cc:3032] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:17.022794 24115 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.023229 23968 raft_consensus.cc:729] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:17.023893 23968 consensus_queue.cc:260] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.024996 23968 raft_consensus.cc:3037] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.026448 24190 raft_consensus.cc:979] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:17.026898 24189 raft_consensus.cc:1067] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:17.028600 23967 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:17.029036 24115 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 2.
I20240501 13:55:17.029192 23967 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:17.029556 23967 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.030215 24010 leader_election.cc:302] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:17.030545 23968 raft_consensus.cc:2454] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 2.
I20240501 13:55:17.031133 24189 raft_consensus.cc:2781] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:17.031612 24189 raft_consensus.cc:686] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:17.033010 24189 consensus_queue.cc:237] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.036198 23967 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.037915 23967 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:17.038684 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:17.038805 24115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:17.039322 24044 raft_consensus.cc:3032] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:17.039443 24115 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.039783 24044 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:17.040529 24044 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.041682 24044 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:17.041121 23864 catalog_manager.cc:5484] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 1 to 2, leader changed from 093b61249a29465596a6542c3b6da169 (127.23.70.129) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 2 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:17.047586 24115 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:17.048947 23935 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:17.049240 24044 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:17.049944 24239 raft_consensus.cc:2781] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:17.050419 24239 raft_consensus.cc:686] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:17.051043 24239 consensus_queue.cc:237] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:17.058137 23864 catalog_manager.cc:5484] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:17.112138 24044 raft_consensus.cc:1261] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.113305 24239 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.121542 24115 raft_consensus.cc:1261] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.123256 24263 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:17.288928 23967 raft_consensus.cc:1261] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.290215 24252 consensus_queue.cc:1035] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.299211 24044 raft_consensus.cc:1261] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.300778 24266 consensus_queue.cc:1035] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:17.406948 24044 raft_consensus.cc:1261] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.408135 24263 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.416165 24115 raft_consensus.cc:1261] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.417670 24239 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.530561 24044 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.531832 24239 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.540813 24115 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.542548 24250 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.626439 23967 raft_consensus.cc:1261] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.627218 24115 raft_consensus.cc:1261] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:17.627774 24268 consensus_queue.cc:1035] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:17.628667 24189 consensus_queue.cc:1035] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:19.633945 23834 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240501 13:55:19.634723 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:19.655645 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:19.656204 24044 raft_consensus.cc:595] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:19.667776 23967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "093b61249a29465596a6542c3b6da169"
mode: GRACEFUL
new_leader_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:48596
I20240501 13:55:19.668309 23967 raft_consensus.cc:595] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Received request to transfer leadership to TS 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:19.669396 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 13:55:19.669871 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:20.002157 24279 raft_consensus.cc:979] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: : Instructing follower 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:20.002601 24263 raft_consensus.cc:1067] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Signalling peer 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:20.003849 24115 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:55518
I20240501 13:55:20.004393 24115 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:20.004712 24115 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.009582 24115 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:20.011633 24115 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903)
I20240501 13:55:20.012403 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:20.012723 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:20.012923 24044 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.013229 23967 raft_consensus.cc:3032] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:20.013489 23967 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:20.014102 23967 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:20.014971 23967 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.017429 24044 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:20.018448 24081 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:20.019104 24258 raft_consensus.cc:2781] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:20.020274 24258 raft_consensus.cc:686] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Becoming Leader. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:20.020404 23967 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:20.021251 24258 consensus_queue.cc:237] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:20.028630 23864 catalog_manager.cc:5484] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d reported cstate change: term changed from 1 to 2, leader changed from 093b61249a29465596a6542c3b6da169 (127.23.70.129) to 06a90d5210104a7b802c94cbe78f375d (127.23.70.131). New cstate: current_term: 2 leader_uuid: "06a90d5210104a7b802c94cbe78f375d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:20.038595 24285 raft_consensus.cc:979] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: : Instructing follower 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:20.039141 24268 raft_consensus.cc:1067] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:20.040766 24115 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:55502
I20240501 13:55:20.041334 24115 raft_consensus.cc:482] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:20.041659 24115 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.046685 24115 raft_consensus.cc:504] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:20.048308 24115 leader_election.cc:288] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:20.049139 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:20.049382 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:20.049647 23967 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.049955 24044 raft_consensus.cc:3032] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:20.050222 24044 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:20.050904 24044 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:20.052120 24044 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:20.055455 23967 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:20.056483 24082 leader_election.cc:302] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:20.056999 24044 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 2.
I20240501 13:55:20.057157 24258 raft_consensus.cc:2781] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:20.057659 24258 raft_consensus.cc:686] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Becoming Leader. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:20.058478 24258 consensus_queue.cc:237] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:20.064854 23864 catalog_manager.cc:5484] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 06a90d5210104a7b802c94cbe78f375d (127.23.70.131). New cstate: current_term: 2 leader_uuid: "06a90d5210104a7b802c94cbe78f375d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:20.552165 23967 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:20.553400 24258 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:20.563798 24044 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:20.567165 24291 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:20.602324 23967 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:20.603427 24289 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:20.611398 24044 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:20.612860 24289 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:20.670765 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:20.675782 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:20.676213 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:21.677152 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:21.682801 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:21.683259 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:22.683984 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:22.688874 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:22.689240 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:23.690037 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:23.694886 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:23.695348 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:24.696075 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:24.700697 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:24.701099 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:25.701858 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:25.706518 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:25.706961 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:26.707715 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:26.712584 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:26.713008 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:27.779309 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:27.945582 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:28.139398 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240501 13:55:28.627067 24065 debug-util.cc:398] Leaking SignalData structure 0x7b0800130060 after lost signal to thread 23837
W20240501 13:55:28.628203 24065 debug-util.cc:398] Leaking SignalData structure 0x7b0800130740 after lost signal to thread 23900
W20240501 13:55:28.629437 24065 debug-util.cc:398] Leaking SignalData structure 0x7b080014f140 after lost signal to thread 23993
W20240501 13:55:28.630339 24065 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7d00 after lost signal to thread 24068
W20240501 13:55:28.631247 24065 debug-util.cc:398] Leaking SignalData structure 0x7b08001ab640 after lost signal to thread 24142
I20240501 13:55:29.140815 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:29.145779 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:29.146204 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:30.174865 23834 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240501 13:55:30.197117 24118 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:40770
I20240501 13:55:30.197949 24118 raft_consensus.cc:595] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:30.203986 23967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2"
dest_uuid: "093b61249a29465596a6542c3b6da169"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:49292
I20240501 13:55:30.204473 23967 raft_consensus.cc:595] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:30.209614 23967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "093b61249a29465596a6542c3b6da169"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:49292
I20240501 13:55:30.210096 23967 raft_consensus.cc:595] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:30.215653 23967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477"
dest_uuid: "093b61249a29465596a6542c3b6da169"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:49292
I20240501 13:55:30.216142 23967 raft_consensus.cc:595] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:30.230163 24118 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
mode: GRACEFUL
new_leader_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:40770
I20240501 13:55:30.230612 24118 raft_consensus.cc:595] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Received request to transfer leadership to TS a22335d124c946bda7b03079b5661572
I20240501 13:55:30.488557 24315 raft_consensus.cc:979] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.489166 24307 raft_consensus.cc:1067] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.491598 24043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57440
I20240501 13:55:30.492237 24043 raft_consensus.cc:482] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:30.492765 24043 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.501032 24043 raft_consensus.cc:504] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.504237 24316 raft_consensus.cc:979] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.504868 24311 raft_consensus.cc:1067] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.506605 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:30.507609 23967 raft_consensus.cc:3032] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:30.507691 24044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57432
I20240501 13:55:30.507982 23967 raft_consensus.cc:729] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:30.507717 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2c504bf8264d589bce4c3c4d29b477" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:30.508708 24044 raft_consensus.cc:482] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:30.508929 24118 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.509119 24044 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.509228 23967 consensus_queue.cc:260] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.510520 23967 raft_consensus.cc:3037] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.516798 24044 raft_consensus.cc:504] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.517570 23967 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.518764 24118 raft_consensus.cc:2454] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.519282 24008 leader_election.cc:302] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:30.519860 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:30.520382 24044 leader_election.cc:288] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:30.521174 23969 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.521492 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:30.522243 24118 raft_consensus.cc:3032] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:30.522433 24118 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:30.523085 24118 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.523876 24118 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.520648 24268 raft_consensus.cc:2781] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:30.505980 24043 leader_election.cc:288] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:30.530151 23969 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.531728 24268 raft_consensus.cc:686] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:30.532186 24008 leader_election.cc:302] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:30.533160 24317 raft_consensus.cc:2781] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:30.532707 24268 consensus_queue.cc:237] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.537959 24316 raft_consensus.cc:979] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.538617 24118 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.538666 24311 raft_consensus.cc:1067] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.541164 24044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57432
I20240501 13:55:30.541802 24044 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:30.542217 24044 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.551008 24044 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:30.554497 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:30.555239 24118 raft_consensus.cc:3032] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:30.555603 24118 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:30.556406 24118 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:30.557662 24118 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.559413 24317 raft_consensus.cc:686] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:30.560454 24317 consensus_queue.cc:237] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.564150 23865 catalog_manager.cc:5484] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 2 to 3, leader changed from 093b61249a29465596a6542c3b6da169 (127.23.70.129) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 3 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:30.569164 24118 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.570538 24010 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:30.571992 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:30.572674 23967 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.575811 24044 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Requested vote from peers 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875), 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903)
I20240501 13:55:30.576068 24317 raft_consensus.cc:2781] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:30.577366 24317 raft_consensus.cc:686] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:30.578570 24317 consensus_queue.cc:237] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:30.580951 23967 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.597643 23865 catalog_manager.cc:5484] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 2 to 3, leader changed from 06a90d5210104a7b802c94cbe78f375d (127.23.70.131) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 3 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:30.598907 23865 catalog_manager.cc:5484] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 2 to 3, leader changed from 06a90d5210104a7b802c94cbe78f375d (127.23.70.131) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 3 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:30.626891 24315 raft_consensus.cc:979] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.627486 24307 raft_consensus.cc:1067] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.630187 24044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57440
I20240501 13:55:30.630762 24044 raft_consensus.cc:482] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:30.631126 24044 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.637579 24044 raft_consensus.cc:504] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.639014 24044 leader_election.cc:288] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:30.640012 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:30.640295 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e54c75c9d07413a9df84bea534f69f2" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:30.640653 23967 raft_consensus.cc:3032] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:30.640766 24118 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.641021 23967 raft_consensus.cc:729] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:30.641737 23967 consensus_queue.cc:260] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.643010 23967 raft_consensus.cc:3037] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.645099 24118 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.646111 24010 leader_election.cc:302] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:30.646764 24317 raft_consensus.cc:2781] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:30.647199 24317 raft_consensus.cc:686] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:30.648289 24317 consensus_queue.cc:237] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.649789 23967 raft_consensus.cc:2454] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.656095 23865 catalog_manager.cc:5484] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 2 to 3, leader changed from 093b61249a29465596a6542c3b6da169 (127.23.70.129) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 3 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:30.713424 24315 raft_consensus.cc:979] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: : Instructing follower a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.713963 24307 raft_consensus.cc:1067] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Signalling peer a22335d124c946bda7b03079b5661572 to start an election
I20240501 13:55:30.715391 24044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "a22335d124c946bda7b03079b5661572"
from {username='slave'} at 127.0.0.1:57440
I20240501 13:55:30.715855 24044 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:30.716105 24044 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.720283 24044 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.721777 24044 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:30.722851 23967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:30.722860 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "a22335d124c946bda7b03079b5661572" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:30.723397 23967 raft_consensus.cc:3032] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:30.723542 24118 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.723666 23967 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:30.724447 23967 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.725276 23967 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:30.729244 23967 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.730238 24008 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:30.730897 24317 raft_consensus.cc:2781] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:30.730890 24118 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22335d124c946bda7b03079b5661572 in term 3.
I20240501 13:55:30.731387 24317 raft_consensus.cc:686] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Leader. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:30.733059 24317 consensus_queue.cc:237] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:30.741318 23865 catalog_manager.cc:5484] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 reported cstate change: term changed from 2 to 3, leader changed from 093b61249a29465596a6542c3b6da169 (127.23.70.129) to a22335d124c946bda7b03079b5661572 (127.23.70.130). New cstate: current_term: 3 leader_uuid: "a22335d124c946bda7b03079b5661572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:30.947922 23967 raft_consensus.cc:1261] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:30.949198 24317 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:30.954433 24118 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:30.955981 24318 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:30.961460 24116 raft_consensus.cc:1261] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:30.962934 24318 consensus_queue.cc:1035] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:30.976272 23967 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:30.978176 24317 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.075994 24116 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.077914 24320 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.086319 23967 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.088078 24317 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.176313 23969 raft_consensus.cc:1261] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.177654 24318 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.188470 24116 raft_consensus.cc:1261] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.189955 24318 consensus_queue.cc:1035] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.215571 24116 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.216857 24318 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:31.225801 23969 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer a22335d124c946bda7b03079b5661572: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:31.227147 24317 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:33.260044 23834 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240501 13:55:33.260896 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:33.267706 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:33.268462 24044 raft_consensus.cc:595] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:33.271392 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "836a46d915f6442eb009a671446afc63"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:33.271847 24044 raft_consensus.cc:595] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Received request to transfer leadership to TS 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:33.273716 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:33.274135 24044 raft_consensus.cc:595] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:33.275955 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:33.276439 24044 raft_consensus.cc:595] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Received request to transfer leadership to TS 06a90d5210104a7b802c94cbe78f375d
I20240501 13:55:33.278498 24044 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4"
dest_uuid: "a22335d124c946bda7b03079b5661572"
mode: GRACEFUL
new_leader_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:54712
I20240501 13:55:33.278966 24044 raft_consensus.cc:595] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Received request to transfer leadership to TS 093b61249a29465596a6542c3b6da169
I20240501 13:55:33.280012 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 13:55:33.280634 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:33.287364 24343 raft_consensus.cc:979] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: : Instructing follower 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:33.287801 24343 raft_consensus.cc:1067] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Signalling peer 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:33.289240 24118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:55502
I20240501 13:55:33.289647 24118 raft_consensus.cc:482] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:33.290007 24118 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.295339 24118 raft_consensus.cc:504] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:33.297277 24118 leader_election.cc:288] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 4 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903)
I20240501 13:55:33.298291 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:33.298403 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7286b7bc0974727ae5650de0f78a3b3" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:33.299041 24044 raft_consensus.cc:3032] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:55:33.299188 23968 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.299319 24044 raft_consensus.cc:729] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:33.299983 24044 consensus_queue.cc:260] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:33.300966 24044 raft_consensus.cc:3037] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.304322 23968 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 4.
I20240501 13:55:33.305557 24082 leader_election.cc:302] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:33.305634 24044 raft_consensus.cc:2454] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 4.
I20240501 13:55:33.306421 24316 raft_consensus.cc:2781] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:55:33.307386 24316 raft_consensus.cc:686] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 4 LEADER]: Becoming Leader. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:33.308104 24316 consensus_queue.cc:237] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } }
I20240501 13:55:33.314939 23864 catalog_manager.cc:5484] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d reported cstate change: term changed from 3 to 4, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 06a90d5210104a7b802c94cbe78f375d (127.23.70.131). New cstate: current_term: 4 leader_uuid: "06a90d5210104a7b802c94cbe78f375d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:33.391278 24320 raft_consensus.cc:979] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.391677 24343 raft_consensus.cc:1067] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.393041 23968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:33.393657 23968 raft_consensus.cc:482] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:33.394021 23968 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:33.398638 23968 raft_consensus.cc:504] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.400655 23968 leader_election.cc:288] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:33.401624 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:33.401660 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f59ff8d5e98045a093914244a83d1ce4" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:33.402210 24044 raft_consensus.cc:3032] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:33.402266 24118 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:33.402452 24044 raft_consensus.cc:729] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:33.403100 24044 consensus_queue.cc:260] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.403942 24044 raft_consensus.cc:3037] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:33.406733 24118 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:33.407795 23935 leader_election.cc:302] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:33.408439 24044 raft_consensus.cc:2454] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 2.
I20240501 13:55:33.409668 24350 raft_consensus.cc:2781] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:33.410832 24350 raft_consensus.cc:686] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:33.411574 24350 consensus_queue.cc:237] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.418011 23864 catalog_manager.cc:5484] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 1 to 2, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:33.544693 24320 raft_consensus.cc:979] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: : Instructing follower 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:33.545128 24343 raft_consensus.cc:1067] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Signalling peer 06a90d5210104a7b802c94cbe78f375d to start an election
I20240501 13:55:33.546449 24118 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "836a46d915f6442eb009a671446afc63"
dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
from {username='slave'} at 127.0.0.1:55502
I20240501 13:55:33.546905 24118 raft_consensus.cc:482] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:33.547160 24118 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:33.551883 24118 raft_consensus.cc:504] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.553267 24118 leader_election.cc:288] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 3 election: Requested vote from peers 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903), a22335d124c946bda7b03079b5661572 (127.23.70.130:36467)
I20240501 13:55:33.554255 23968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "093b61249a29465596a6542c3b6da169"
I20240501 13:55:33.554270 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836a46d915f6442eb009a671446afc63" candidate_uuid: "06a90d5210104a7b802c94cbe78f375d" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:33.554837 23968 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:33.554925 24044 raft_consensus.cc:3032] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:33.555240 24044 raft_consensus.cc:729] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:33.555933 24044 consensus_queue.cc:260] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.556825 24044 raft_consensus.cc:3037] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:33.559459 23968 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 3.
I20240501 13:55:33.560453 24082 leader_election.cc:302] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06a90d5210104a7b802c94cbe78f375d, 093b61249a29465596a6542c3b6da169; no voters:
I20240501 13:55:33.561167 24044 raft_consensus.cc:2454] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06a90d5210104a7b802c94cbe78f375d in term 3.
I20240501 13:55:33.561214 24316 raft_consensus.cc:2781] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:33.561774 24316 raft_consensus.cc:686] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 3 LEADER]: Becoming Leader. State: Replica: 06a90d5210104a7b802c94cbe78f375d, State: Running, Role: LEADER
I20240501 13:55:33.562594 24316 consensus_queue.cc:237] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.569090 23864 catalog_manager.cc:5484] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d reported cstate change: term changed from 2 to 3, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 06a90d5210104a7b802c94cbe78f375d (127.23.70.131). New cstate: current_term: 3 leader_uuid: "06a90d5210104a7b802c94cbe78f375d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:33.638518 24320 raft_consensus.cc:979] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.639048 24343 raft_consensus.cc:1067] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.640419 23968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:33.640889 23968 raft_consensus.cc:482] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:33.641160 23968 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.676712 23968 raft_consensus.cc:504] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.678303 23968 leader_election.cc:288] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 4 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:33.679349 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:33.679307 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122b6f8fb7ab4b89bf8751df210ceaaa" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:33.679961 24044 raft_consensus.cc:3032] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:55:33.680011 24118 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.680243 24044 raft_consensus.cc:729] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:33.681022 24044 consensus_queue.cc:260] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.682010 24044 raft_consensus.cc:3037] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.708786 24044 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 4.
I20240501 13:55:33.708786 24118 raft_consensus.cc:2454] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 4.
I20240501 13:55:33.710218 23932 leader_election.cc:302] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:33.711138 24350 raft_consensus.cc:2781] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:55:33.711649 24350 raft_consensus.cc:686] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 4 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:33.712704 24350 consensus_queue.cc:237] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.720176 23864 catalog_manager.cc:5484] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 3 to 4, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 4 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:33.724476 24044 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:33.726011 24316 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:33.726855 23968 raft_consensus.cc:1261] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:33.728685 24316 consensus_queue.cc:1035] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:33.837510 24044 raft_consensus.cc:1261] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:33.838919 24350 consensus_queue.cc:1035] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:33.849063 24118 raft_consensus.cc:1261] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:33.850883 24361 consensus_queue.cc:1035] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:33.857044 24320 raft_consensus.cc:979] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: : Instructing follower 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.860251 24364 raft_consensus.cc:1067] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Signalling peer 093b61249a29465596a6542c3b6da169 to start an election
I20240501 13:55:33.864688 23968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2"
dest_uuid: "093b61249a29465596a6542c3b6da169"
from {username='slave'} at 127.0.0.1:49284
I20240501 13:55:33.865350 23968 raft_consensus.cc:482] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:33.865625 23968 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.899801 23968 raft_consensus.cc:504] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.901424 23968 leader_election.cc:288] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 4 election: Requested vote from peers a22335d124c946bda7b03079b5661572 (127.23.70.130:36467), 06a90d5210104a7b802c94cbe78f375d (127.23.70.131:35875)
I20240501 13:55:33.902310 24044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a22335d124c946bda7b03079b5661572"
I20240501 13:55:33.902612 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac44a39a04d458badaf6ced34c24bc2" candidate_uuid: "093b61249a29465596a6542c3b6da169" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "06a90d5210104a7b802c94cbe78f375d"
I20240501 13:55:33.902915 24044 raft_consensus.cc:3032] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:55:33.903255 24044 raft_consensus.cc:729] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a22335d124c946bda7b03079b5661572, State: Running, Role: LEADER
I20240501 13:55:33.903273 24118 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.903905 24044 consensus_queue.cc:260] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:33.904793 24044 raft_consensus.cc:3037] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:33.996124 24118 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 4.
I20240501 13:55:33.996124 24044 raft_consensus.cc:2454] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 093b61249a29465596a6542c3b6da169 in term 4.
I20240501 13:55:33.997992 23932 leader_election.cc:302] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 093b61249a29465596a6542c3b6da169, a22335d124c946bda7b03079b5661572; no voters:
I20240501 13:55:33.999053 24361 raft_consensus.cc:2781] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:55:33.999553 24361 raft_consensus.cc:686] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 4 LEADER]: Becoming Leader. State: Replica: 093b61249a29465596a6542c3b6da169, State: Running, Role: LEADER
I20240501 13:55:34.000540 24361 consensus_queue.cc:237] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } }
I20240501 13:55:34.008473 23866 catalog_manager.cc:5484] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 reported cstate change: term changed from 3 to 4, leader changed from a22335d124c946bda7b03079b5661572 (127.23.70.130) to 093b61249a29465596a6542c3b6da169 (127.23.70.129). New cstate: current_term: 4 leader_uuid: "093b61249a29465596a6542c3b6da169" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:34.069849 23968 raft_consensus.cc:1261] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 13:55:34.071213 24316 consensus_queue.cc:1035] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "093b61249a29465596a6542c3b6da169" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:34.079525 24044 raft_consensus.cc:1261] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Refusing update from remote peer 06a90d5210104a7b802c94cbe78f375d: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 13:55:34.081183 24344 consensus_queue.cc:1035] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:34.189909 24118 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:34.191145 24361 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:34.201236 24044 raft_consensus.cc:1261] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:34.202908 24370 consensus_queue.cc:1035] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:55:34.281626 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:34.286711 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:34.287134 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:34.385447 24116 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:34.386698 24350 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a90d5210104a7b802c94cbe78f375d" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:34.396028 24044 raft_consensus.cc:1261] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Refusing update from remote peer 093b61249a29465596a6542c3b6da169: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:34.398373 24361 consensus_queue.cc:1035] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22335d124c946bda7b03079b5661572" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 36467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:35.287925 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:35.293367 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:35.293779 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:36.294534 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:36.299535 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:36.299970 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:37.300838 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:37.305790 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:37.306210 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:38.307014 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:38.311951 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:38.312393 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:39.313172 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:39.317960 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:39.318375 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:40.319206 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:40.324605 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:40.325030 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:41.325768 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:41.330654 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:41.331069 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:42.332118 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:42.337193 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:42.337628 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:43.361896 23834 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240501 13:55:43.362480 23834 tablet_server.cc:178] TabletServer@127.23.70.129:0 shutting down...
I20240501 13:55:43.390846 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:55:43.391561 23834 tablet_replica.cc:330] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.392619 23834 raft_consensus.cc:2229] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 4 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.393654 23834 raft_consensus.cc:2258] T 9ac44a39a04d458badaf6ced34c24bc2 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.396154 23834 tablet_replica.cc:330] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.396698 23834 raft_consensus.cc:2229] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.397135 23834 raft_consensus.cc:2258] T bd2c504bf8264d589bce4c3c4d29b477 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.398715 23834 tablet_replica.cc:330] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.399183 23834 raft_consensus.cc:2229] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.399580 23834 raft_consensus.cc:2258] T bdb52db7fa1340c68d6845e53b1b5e49 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.401382 23834 tablet_replica.cc:330] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.401854 23834 raft_consensus.cc:2229] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.402637 23834 raft_consensus.cc:2258] T f59ff8d5e98045a093914244a83d1ce4 P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.404296 23834 tablet_replica.cc:330] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.404817 23834 raft_consensus.cc:2229] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.405227 23834 raft_consensus.cc:2258] T d7286b7bc0974727ae5650de0f78a3b3 P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.406829 23834 tablet_replica.cc:330] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.407326 23834 raft_consensus.cc:2229] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.407720 23834 raft_consensus.cc:2258] T 48cb8a2eeba14b759fb7591a5793a99d P 093b61249a29465596a6542c3b6da169 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.409456 23834 tablet_replica.cc:330] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.409977 23834 raft_consensus.cc:2229] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 4 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.410761 23834 raft_consensus.cc:2258] T 122b6f8fb7ab4b89bf8751df210ceaaa P 093b61249a29465596a6542c3b6da169 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.412499 23834 tablet_replica.cc:330] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.413040 23834 raft_consensus.cc:2229] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.413430 23834 raft_consensus.cc:2258] T 5e54c75c9d07413a9df84bea534f69f2 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.415146 23834 tablet_replica.cc:330] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169: stopping tablet replica
I20240501 13:55:43.415571 23834 raft_consensus.cc:2229] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.415953 23834 raft_consensus.cc:2258] T 836a46d915f6442eb009a671446afc63 P 093b61249a29465596a6542c3b6da169 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240501 13:55:43.504583 24082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.70.129:35903: connect: Connection refused (error 111)
I20240501 13:55:43.507689 23834 tablet_server.cc:194] TabletServer@127.23.70.129:0 shutdown complete.
W20240501 13:55:43.508829 24008 consensus_peers.cc:485] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 -> Peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Couldn't send request to peer 093b61249a29465596a6542c3b6da169. Status: Network error: Client connection negotiation failed: client connection to 127.23.70.129:35903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 13:55:43.509536 24082 consensus_peers.cc:485] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d -> Peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Couldn't send request to peer 093b61249a29465596a6542c3b6da169. Status: Network error: Client connection negotiation failed: client connection to 127.23.70.129:35903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:55:43.526618 23834 tablet_server.cc:178] TabletServer@127.23.70.130:0 shutting down...
W20240501 13:55:43.535940 24081 consensus_peers.cc:485] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d -> Peer a22335d124c946bda7b03079b5661572 (127.23.70.130:36467): Couldn't send request to peer a22335d124c946bda7b03079b5661572. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:55:43.557901 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:55:43.558611 23834 tablet_replica.cc:330] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.559274 23834 raft_consensus.cc:2229] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.559716 23834 raft_consensus.cc:2258] T 48cb8a2eeba14b759fb7591a5793a99d P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.561522 23834 tablet_replica.cc:330] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.562075 23834 raft_consensus.cc:2229] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.562886 23834 raft_consensus.cc:2258] T bd2c504bf8264d589bce4c3c4d29b477 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.564574 23834 tablet_replica.cc:330] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.565038 23834 raft_consensus.cc:2229] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.565762 23834 raft_consensus.cc:2258] T bdb52db7fa1340c68d6845e53b1b5e49 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.567332 23834 tablet_replica.cc:330] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.567791 23834 raft_consensus.cc:2229] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.568166 23834 raft_consensus.cc:2258] T f59ff8d5e98045a093914244a83d1ce4 P a22335d124c946bda7b03079b5661572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.569759 23834 tablet_replica.cc:330] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.570288 23834 raft_consensus.cc:2229] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.570677 23834 raft_consensus.cc:2258] T 836a46d915f6442eb009a671446afc63 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.572630 23834 tablet_replica.cc:330] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.573271 23834 raft_consensus.cc:2229] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.573750 23834 raft_consensus.cc:2258] T 9ac44a39a04d458badaf6ced34c24bc2 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.576059 23834 tablet_replica.cc:330] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.576541 23834 raft_consensus.cc:2229] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.576930 23834 raft_consensus.cc:2258] T 122b6f8fb7ab4b89bf8751df210ceaaa P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.578398 23834 tablet_replica.cc:330] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.578801 23834 raft_consensus.cc:2229] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.579550 23834 raft_consensus.cc:2258] T 5e54c75c9d07413a9df84bea534f69f2 P a22335d124c946bda7b03079b5661572 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.581274 23834 tablet_replica.cc:330] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572: stopping tablet replica
I20240501 13:55:43.581715 23834 raft_consensus.cc:2229] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.582123 23834 raft_consensus.cc:2258] T d7286b7bc0974727ae5650de0f78a3b3 P a22335d124c946bda7b03079b5661572 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.626993 23834 tablet_server.cc:194] TabletServer@127.23.70.130:0 shutdown complete.
W20240501 13:55:43.636260 24082 consensus_peers.cc:485] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d -> Peer 093b61249a29465596a6542c3b6da169 (127.23.70.129:35903): Couldn't send request to peer 093b61249a29465596a6542c3b6da169. Status: Network error: Client connection negotiation failed: client connection to 127.23.70.129:35903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:55:43.645339 23834 tablet_server.cc:178] TabletServer@127.23.70.131:0 shutting down...
I20240501 13:55:43.674918 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:55:43.676680 23834 tablet_replica.cc:330] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.677718 23834 raft_consensus.cc:2229] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.679140 23834 raft_consensus.cc:2258] T 48cb8a2eeba14b759fb7591a5793a99d P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.681479 23834 tablet_replica.cc:330] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.682181 23834 raft_consensus.cc:2229] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.683024 23834 raft_consensus.cc:2258] T 836a46d915f6442eb009a671446afc63 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.684649 23834 tablet_replica.cc:330] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.685143 23834 raft_consensus.cc:2229] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.685528 23834 raft_consensus.cc:2258] T bd2c504bf8264d589bce4c3c4d29b477 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.687076 23834 tablet_replica.cc:330] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.687536 23834 raft_consensus.cc:2229] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.687950 23834 raft_consensus.cc:2258] T bdb52db7fa1340c68d6845e53b1b5e49 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.689587 23834 tablet_replica.cc:330] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.690116 23834 raft_consensus.cc:2229] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.690629 23834 raft_consensus.cc:2258] T f59ff8d5e98045a093914244a83d1ce4 P 06a90d5210104a7b802c94cbe78f375d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.692301 23834 tablet_replica.cc:330] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.692809 23834 raft_consensus.cc:2229] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 4 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.693584 23834 raft_consensus.cc:2258] T d7286b7bc0974727ae5650de0f78a3b3 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.695205 23834 tablet_replica.cc:330] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.695693 23834 raft_consensus.cc:2229] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.696094 23834 raft_consensus.cc:2258] T 9ac44a39a04d458badaf6ced34c24bc2 P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.697753 23834 tablet_replica.cc:330] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.698195 23834 raft_consensus.cc:2229] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.698573 23834 raft_consensus.cc:2258] T 5e54c75c9d07413a9df84bea534f69f2 P 06a90d5210104a7b802c94cbe78f375d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.700313 23834 tablet_replica.cc:330] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d: stopping tablet replica
I20240501 13:55:43.700836 23834 raft_consensus.cc:2229] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:55:43.701252 23834 raft_consensus.cc:2258] T 122b6f8fb7ab4b89bf8751df210ceaaa P 06a90d5210104a7b802c94cbe78f375d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.728883 23834 tablet_server.cc:194] TabletServer@127.23.70.131:0 shutdown complete.
I20240501 13:55:43.746445 23834 master.cc:537] Master@127.23.70.190:42367 shutting down...
I20240501 13:55:43.903554 23834 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:55:43.904199 23834 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:55:43.904551 23834 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f574cbd1a4d41f3985dcb9508e980db: stopping tablet replica
I20240501 13:55:43.922258 23834 master.cc:552] Master@127.23.70.190:42367 shutdown complete.
06a90d5210104a7b802c94cbe78f375d 1
093b61249a29465596a6542c3b6da169 4
a22335d124c946bda7b03079b5661572 4
06a90d5210104a7b802c94cbe78f375d 3
093b61249a29465596a6542c3b6da169 3
a22335d124c946bda7b03079b5661572 3
06a90d5210104a7b802c94cbe78f375d 1
093b61249a29465596a6542c3b6da169 0
a22335d124c946bda7b03079b5661572 8
06a90d5210104a7b802c94cbe78f375d 3
093b61249a29465596a6542c3b6da169 3
a22335d124c946bda7b03079b5661572 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32651 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240501 13:55:43.963022 23834 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.70.190:44121
I20240501 13:55:43.964154 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:43.970629 24383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:43.971558 24384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:43.972828 23834 server_base.cc:1034] running on GCE node
W20240501 13:55:43.973726 24386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:43.974659 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:43.974879 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:43.975019 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571743975003 us; error 0 us; skew 500 ppm
I20240501 13:55:43.975507 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:43.982913 23834 webserver.cc:460] Webserver started at http://127.23.70.190:43541/ using document root <none> and password file <none>
I20240501 13:55:43.983440 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:43.983623 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:43.983906 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:43.985144 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/master-0-root/instance:
uuid: "a9a0c08db05244b4bdc9d4787d7acea7"
format_stamp: "Formatted at 2024-05-01 13:55:43 on dist-test-slave-s72h"
I20240501 13:55:43.989754 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:55:43.993296 24391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:43.994158 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:55:43.994416 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/master-0-root
uuid: "a9a0c08db05244b4bdc9d4787d7acea7"
format_stamp: "Formatted at 2024-05-01 13:55:43 on dist-test-slave-s72h"
I20240501 13:55:43.994676 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:44.007428 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:44.008679 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:44.046618 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.190:44121
I20240501 13:55:44.046721 24442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.190:44121 every 8 connection(s)
I20240501 13:55:44.050882 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.060575 24443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7: Bootstrap starting.
I20240501 13:55:44.064910 24443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.069307 24443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7: No bootstrap required, opened a new log
I20240501 13:55:44.071420 24443 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER }
I20240501 13:55:44.071908 24443 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.072163 24443 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a0c08db05244b4bdc9d4787d7acea7, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.072763 24443 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER }
I20240501 13:55:44.073232 24443 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:55:44.073460 24443 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:55:44.073719 24443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:44.078003 24443 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER }
I20240501 13:55:44.078557 24443 leader_election.cc:302] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a0c08db05244b4bdc9d4787d7acea7; no voters:
I20240501 13:55:44.079679 24443 leader_election.cc:288] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:55:44.080129 24446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:44.081427 24446 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 1 LEADER]: Becoming Leader. State: Replica: a9a0c08db05244b4bdc9d4787d7acea7, State: Running, Role: LEADER
I20240501 13:55:44.082178 24446 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER }
I20240501 13:55:44.082721 24443 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:55:44.086841 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a0c08db05244b4bdc9d4787d7acea7. Latest consensus state: current_term: 1 leader_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER } }
I20240501 13:55:44.087671 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a0c08db05244b4bdc9d4787d7acea7" member_type: VOTER } }
I20240501 13:55:44.087903 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [sys.catalog]: This master's current role is: LEADER
I20240501 13:55:44.088783 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [sys.catalog]: This master's current role is: LEADER
I20240501 13:55:44.090917 24454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:55:44.098035 24454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:55:44.099005 23834 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:55:44.107014 24454 catalog_manager.cc:1347] Generated new cluster ID: d9402355f9564bdf9b04e5a2311b910e
I20240501 13:55:44.107343 24454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:55:44.144418 24454 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:55:44.145815 24454 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:55:44.158716 24454 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7: Generated new TSK 0
I20240501 13:55:44.159570 24454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:55:44.165519 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:44.172325 24464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:44.173965 24465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:44.174508 24467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:44.175563 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:44.176398 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:44.176579 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:44.176697 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571744176686 us; error 0 us; skew 500 ppm
I20240501 13:55:44.177189 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:44.179654 23834 webserver.cc:460] Webserver started at http://127.23.70.129:34829/ using document root <none> and password file <none>
I20240501 13:55:44.180140 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:44.180300 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:44.180598 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:44.181689 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-0-root/instance:
uuid: "bbaf3058782245828d49c901c40f1399"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.186349 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:55:44.189661 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.190419 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:55:44.190712 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
uuid: "bbaf3058782245828d49c901c40f1399"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.190981 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:44.212769 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:44.213951 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:44.215390 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:44.217818 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:44.218037 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.218319 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:44.218477 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.257433 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.129:38341
I20240501 13:55:44.257550 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.129:38341 every 8 connection(s)
I20240501 13:55:44.262195 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:44.270500 24539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:44.271132 24540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:44.272197 24542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:44.274221 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:44.275357 24535 heartbeater.cc:347] Connected to a master server at 127.23.70.190:44121
I20240501 13:55:44.275480 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 13:55:44.275693 24535 heartbeater.cc:464] Registering TS with master...
W20240501 13:55:44.275734 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:44.276005 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571744275990 us; error 0 us; skew 500 ppm
I20240501 13:55:44.276618 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:44.276561 24535 heartbeater.cc:511] Master 127.23.70.190:44121 requested a full tablet report, sending...
I20240501 13:55:44.279052 24408 ts_manager.cc:194] Registered new tserver with Master: bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341)
I20240501 13:55:44.279467 23834 webserver.cc:460] Webserver started at http://127.23.70.130:39167/ using document root <none> and password file <none>
I20240501 13:55:44.280164 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:44.280441 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:44.280795 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:44.281515 24408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51988
I20240501 13:55:44.282334 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-1-root/instance:
uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.287075 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 13:55:44.291633 24547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.292518 23834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:55:44.292858 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.293164 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:44.317387 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:44.318748 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:44.320185 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:44.322459 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:44.322676 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.322911 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:44.323065 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.362360 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.130:33689
I20240501 13:55:44.362459 24609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.130:33689 every 8 connection(s)
I20240501 13:55:44.367390 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:55:44.375288 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:55:44.378618 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:44.379582 24610 heartbeater.cc:347] Connected to a master server at 127.23.70.190:44121
I20240501 13:55:44.380249 24610 heartbeater.cc:464] Registering TS with master...
I20240501 13:55:44.381237 24610 heartbeater.cc:511] Master 127.23.70.190:44121 requested a full tablet report, sending...
W20240501 13:55:44.383567 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:55:44.383607 24408 ts_manager.cc:194] Registered new tserver with Master: d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:44.384706 23834 server_base.cc:1034] running on GCE node
I20240501 13:55:44.385308 24408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51992
I20240501 13:55:44.385587 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:55:44.385808 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:55:44.385974 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571744385958 us; error 0 us; skew 500 ppm
I20240501 13:55:44.386687 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:55:44.389369 23834 webserver.cc:460] Webserver started at http://127.23.70.131:42467/ using document root <none> and password file <none>
I20240501 13:55:44.389856 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:55:44.390018 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:55:44.390234 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:55:44.391391 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-2-root/instance:
uuid: "2ad4fb9054d644748b36aac4798f724e"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.396167 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 13:55:44.399581 24621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.400440 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:55:44.400763 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
uuid: "2ad4fb9054d644748b36aac4798f724e"
format_stamp: "Formatted at 2024-05-01 13:55:44 on dist-test-slave-s72h"
I20240501 13:55:44.401031 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714571711146223-23834-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:55:44.413090 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:55:44.414283 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:55:44.415762 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:55:44.418119 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:55:44.418313 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.418572 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:55:44.418780 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:55:44.458027 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.131:33637
I20240501 13:55:44.458113 24683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.131:33637 every 8 connection(s)
I20240501 13:55:44.471026 24684 heartbeater.cc:347] Connected to a master server at 127.23.70.190:44121
I20240501 13:55:44.471393 24684 heartbeater.cc:464] Registering TS with master...
I20240501 13:55:44.472052 24684 heartbeater.cc:511] Master 127.23.70.190:44121 requested a full tablet report, sending...
I20240501 13:55:44.473778 24408 ts_manager.cc:194] Registered new tserver with Master: 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:44.474702 23834 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013509697s
I20240501 13:55:44.475062 24408 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52008
I20240501 13:55:44.477224 23834 test_util.cc:268] Using random seed: -674922482
I20240501 13:55:44.499917 24408 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52022:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 13:55:44.503021 24408 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:55:44.574473 24649 tablet_service.cc:1467] Processing CreateTablet for tablet 045ac8899eb64675a94d3955f9816670 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 13:55:44.576050 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045ac8899eb64675a94d3955f9816670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.575942 24648 tablet_service.cc:1467] Processing CreateTablet for tablet 8c83855d66734897831036f82ae28e94 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 13:55:44.576102 24645 tablet_service.cc:1467] Processing CreateTablet for tablet ccd70340327a488aa092093096d4bf1b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 13:55:44.577539 24648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c83855d66734897831036f82ae28e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.578650 24575 tablet_service.cc:1467] Processing CreateTablet for tablet 045ac8899eb64675a94d3955f9816670 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 13:55:44.578938 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 8c83855d66734897831036f82ae28e94 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 13:55:44.581698 24646 tablet_service.cc:1467] Processing CreateTablet for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 13:55:44.582312 24573 tablet_service.cc:1467] Processing CreateTablet for tablet c99207ce180c4f8b91e00bfbe9e9bc6b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 13:55:44.583746 24647 tablet_service.cc:1467] Processing CreateTablet for tablet c99207ce180c4f8b91e00bfbe9e9bc6b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 13:55:44.585626 24572 tablet_service.cc:1467] Processing CreateTablet for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 13:55:44.588218 24646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.588089 24642 tablet_service.cc:1467] Processing CreateTablet for tablet cb834bb158944475a2efb3c109076eed (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 13:55:44.580155 24575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045ac8899eb64675a94d3955f9816670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.589660 24644 tablet_service.cc:1467] Processing CreateTablet for tablet f9992986258f4becb6995d28b5c3127d (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 13:55:44.591446 24640 tablet_service.cc:1467] Processing CreateTablet for tablet 766fa8acf9e14ef2b47b2e9545d31dc4 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 13:55:44.593515 24643 tablet_service.cc:1467] Processing CreateTablet for tablet ab5350250ac144378a3cabb34c92412a (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 13:55:44.592698 24642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb834bb158944475a2efb3c109076eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.594257 24641 tablet_service.cc:1467] Processing CreateTablet for tablet 120f3f71269e4400962f36d05d1ae459 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 13:55:44.595265 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766fa8acf9e14ef2b47b2e9545d31dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.598786 24571 tablet_service.cc:1467] Processing CreateTablet for tablet ccd70340327a488aa092093096d4bf1b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 13:55:44.599891 24568 tablet_service.cc:1467] Processing CreateTablet for tablet cb834bb158944475a2efb3c109076eed (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 13:55:44.601220 24569 tablet_service.cc:1467] Processing CreateTablet for tablet ab5350250ac144378a3cabb34c92412a (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 13:55:44.599929 24570 tablet_service.cc:1467] Processing CreateTablet for tablet f9992986258f4becb6995d28b5c3127d (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 13:55:44.603066 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5350250ac144378a3cabb34c92412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.605123 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99207ce180c4f8b91e00bfbe9e9bc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.605815 24641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120f3f71269e4400962f36d05d1ae459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.606451 24645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd70340327a488aa092093096d4bf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.607160 24644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9992986258f4becb6995d28b5c3127d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.602438 24573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99207ce180c4f8b91e00bfbe9e9bc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.613014 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c83855d66734897831036f82ae28e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.613724 24568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb834bb158944475a2efb3c109076eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.616676 24569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5350250ac144378a3cabb34c92412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.620739 24570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9992986258f4becb6995d28b5c3127d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.624701 24572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.626969 24571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd70340327a488aa092093096d4bf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.629977 24567 tablet_service.cc:1467] Processing CreateTablet for tablet 120f3f71269e4400962f36d05d1ae459 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 13:55:44.641489 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120f3f71269e4400962f36d05d1ae459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.640017 24566 tablet_service.cc:1467] Processing CreateTablet for tablet 766fa8acf9e14ef2b47b2e9545d31dc4 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 13:55:44.649284 24566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766fa8acf9e14ef2b47b2e9545d31dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.661778 24500 tablet_service.cc:1467] Processing CreateTablet for tablet 045ac8899eb64675a94d3955f9816670 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 13:55:44.663247 24500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 045ac8899eb64675a94d3955f9816670. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.665392 24495 tablet_service.cc:1467] Processing CreateTablet for tablet f9992986258f4becb6995d28b5c3127d (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 13:55:44.666795 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9992986258f4becb6995d28b5c3127d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.666666 24494 tablet_service.cc:1467] Processing CreateTablet for tablet ab5350250ac144378a3cabb34c92412a (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 13:55:44.667138 24496 tablet_service.cc:1467] Processing CreateTablet for tablet ccd70340327a488aa092093096d4bf1b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 13:55:44.665510 24499 tablet_service.cc:1467] Processing CreateTablet for tablet 8c83855d66734897831036f82ae28e94 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 13:55:44.668952 24498 tablet_service.cc:1467] Processing CreateTablet for tablet c99207ce180c4f8b91e00bfbe9e9bc6b (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 13:55:44.672581 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5350250ac144378a3cabb34c92412a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.672461 24497 tablet_service.cc:1467] Processing CreateTablet for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 13:55:44.675946 24499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c83855d66734897831036f82ae28e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.676728 24497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5f0c2a27344cb0af1f5b4f8adbb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.680442 24493 tablet_service.cc:1467] Processing CreateTablet for tablet cb834bb158944475a2efb3c109076eed (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 13:55:44.684777 24491 tablet_service.cc:1467] Processing CreateTablet for tablet 766fa8acf9e14ef2b47b2e9545d31dc4 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 13:55:44.686655 24492 tablet_service.cc:1467] Processing CreateTablet for tablet 120f3f71269e4400962f36d05d1ae459 (DEFAULT_TABLE table=test-workload [id=4bef3dad18e24d3d9789eeb7594086a9]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 13:55:44.692585 24498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99207ce180c4f8b91e00bfbe9e9bc6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.697135 24493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb834bb158944475a2efb3c109076eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.700738 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120f3f71269e4400962f36d05d1ae459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.704707 24491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766fa8acf9e14ef2b47b2e9545d31dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.708740 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd70340327a488aa092093096d4bf1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:55:44.753454 24699 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:44.816953 24699 tablet_bootstrap.cc:654] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.819938 24700 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:44.825803 24701 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:44.832830 24701 tablet_bootstrap.cc:654] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.833434 24700 tablet_bootstrap.cc:654] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.834622 24699 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:44.835140 24699 ts_tablet_manager.cc:1367] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.014s
I20240501 13:55:44.837815 24699 raft_consensus.cc:348] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.838585 24699 raft_consensus.cc:374] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.838878 24699 raft_consensus.cc:729] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.839023 24700 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:44.839506 24700 ts_tablet_manager.cc:1367] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20240501 13:55:44.839634 24699 consensus_queue.cc:260] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.842559 24684 heartbeater.cc:503] Master 127.23.70.190:44121 was elected leader, sending a full tablet report...
I20240501 13:55:44.843070 24699 ts_tablet_manager.cc:1398] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 13:55:44.842655 24700 raft_consensus.cc:348] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.843137 24701 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:44.843501 24700 raft_consensus.cc:374] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.843768 24701 ts_tablet_manager.cc:1367] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20240501 13:55:44.843843 24700 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.844115 24699 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:44.844866 24700 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.846735 24701 raft_consensus.cc:348] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.847326 24701 raft_consensus.cc:374] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.847600 24701 raft_consensus.cc:729] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.848435 24701 consensus_queue.cc:260] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.851433 24699 tablet_bootstrap.cc:654] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.857148 24535 heartbeater.cc:503] Master 127.23.70.190:44121 was elected leader, sending a full tablet report...
I20240501 13:55:44.858100 24701 ts_tablet_manager.cc:1398] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20240501 13:55:44.859167 24701 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:44.865561 24701 tablet_bootstrap.cc:654] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.870429 24699 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:44.870999 24699 ts_tablet_manager.cc:1367] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240501 13:55:44.872247 24610 heartbeater.cc:503] Master 127.23.70.190:44121 was elected leader, sending a full tablet report...
I20240501 13:55:44.873884 24699 raft_consensus.cc:348] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.874668 24699 raft_consensus.cc:374] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.874982 24699 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.875674 24699 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.878795 24699 ts_tablet_manager.cc:1398] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20240501 13:55:44.879774 24699 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:44.885877 24699 tablet_bootstrap.cc:654] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.887015 24700 ts_tablet_manager.cc:1398] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.047s user 0.006s sys 0.011s
I20240501 13:55:44.887049 24701 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:44.887727 24701 ts_tablet_manager.cc:1367] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.008s
I20240501 13:55:44.888204 24700 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:44.890585 24701 raft_consensus.cc:348] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.891472 24701 raft_consensus.cc:374] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.891778 24701 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.892629 24701 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.895085 24701 ts_tablet_manager.cc:1398] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 13:55:44.896039 24701 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:44.897657 24700 tablet_bootstrap.cc:654] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.903586 24701 tablet_bootstrap.cc:654] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.903636 24699 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:44.904405 24699 ts_tablet_manager.cc:1367] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.002s
I20240501 13:55:44.908030 24699 raft_consensus.cc:348] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.909127 24699 raft_consensus.cc:374] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.909435 24699 raft_consensus.cc:729] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.910321 24699 consensus_queue.cc:260] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.911942 24700 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:44.912475 24700 ts_tablet_manager.cc:1367] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.010s
I20240501 13:55:44.912566 24699 ts_tablet_manager.cc:1398] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240501 13:55:44.913556 24699 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:44.915282 24701 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:44.915218 24700 raft_consensus.cc:348] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.915948 24701 ts_tablet_manager.cc:1367] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.016s
I20240501 13:55:44.916133 24700 raft_consensus.cc:374] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.916529 24700 raft_consensus.cc:729] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.917007 24707 raft_consensus.cc:482] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:44.917505 24707 raft_consensus.cc:504] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.917657 24700 consensus_queue.cc:260] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.921379 24699 tablet_bootstrap.cc:654] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.923177 24707 leader_election.cc:288] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:44.928407 24700 ts_tablet_manager.cc:1398] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
I20240501 13:55:44.929719 24700 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:44.938089 24700 tablet_bootstrap.cc:654] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.932191 24701 raft_consensus.cc:348] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.941640 24701 raft_consensus.cc:374] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.941985 24701 raft_consensus.cc:729] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.942692 24701 consensus_queue.cc:260] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.945348 24699 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:44.945986 24699 ts_tablet_manager.cc:1367] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.020s
I20240501 13:55:44.948199 24701 ts_tablet_manager.cc:1398] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.032s user 0.003s sys 0.003s
I20240501 13:55:44.949167 24701 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:44.949975 24699 raft_consensus.cc:348] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.951035 24699 raft_consensus.cc:374] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.951391 24699 raft_consensus.cc:729] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.951258 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
W20240501 13:55:44.953076 24473 leader_election.cc:341] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:44.952487 24699 consensus_queue.cc:260] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.956768 24701 tablet_bootstrap.cc:654] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.959051 24699 ts_tablet_manager.cc:1398] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20240501 13:55:44.960151 24699 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
W20240501 13:55:44.963508 24685 tablet.cc:2369] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:55:44.966377 24699 tablet_bootstrap.cc:654] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:44.968278 24701 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:44.968786 24701 ts_tablet_manager.cc:1367] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.009s
I20240501 13:55:44.972970 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:44.972707 24701 raft_consensus.cc:348] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:44.973547 24701 raft_consensus.cc:374] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.973544 24585 raft_consensus.cc:2454] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 0.
I20240501 13:55:44.974006 24701 raft_consensus.cc:729] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.974911 24476 leader_election.cc:302] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e
I20240501 13:55:44.975577 24707 raft_consensus.cc:2781] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:44.975745 24700 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:44.975960 24707 raft_consensus.cc:482] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:44.976006 24701 consensus_queue.cc:260] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:44.976543 24700 ts_tablet_manager.cc:1367] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.047s user 0.020s sys 0.011s
I20240501 13:55:44.976519 24707 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:44.980278 24699 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:44.979876 24700 raft_consensus.cc:348] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.980649 24700 raft_consensus.cc:374] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.980793 24699 ts_tablet_manager.cc:1367] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20240501 13:55:44.980969 24700 raft_consensus.cc:729] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.981815 24700 consensus_queue.cc:260] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.982723 24707 raft_consensus.cc:504] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:44.983523 24699 raft_consensus.cc:348] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:44.984468 24699 raft_consensus.cc:374] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:44.984781 24699 raft_consensus.cc:729] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:44.985596 24699 consensus_queue.cc:260] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:44.986371 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:44.987179 24585 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:44.991307 24699 ts_tablet_manager.cc:1398] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20240501 13:55:44.998613 24700 ts_tablet_manager.cc:1398] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.021s user 0.005s sys 0.002s
I20240501 13:55:45.001370 24585 raft_consensus.cc:2454] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 1.
I20240501 13:55:45.002658 24700 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.005244 24701 ts_tablet_manager.cc:1398] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.036s user 0.011s sys 0.016s
I20240501 13:55:45.007615 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.007927 24701 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
W20240501 13:55:45.009840 24473 leader_election.cc:341] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.011737 24700 tablet_bootstrap.cc:654] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.012014 24720 raft_consensus.cc:482] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.003664 24699 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:45.012483 24720 raft_consensus.cc:504] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.014693 24701 tablet_bootstrap.cc:654] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.015568 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.016306 24659 raft_consensus.cc:2454] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 0.
I20240501 13:55:45.017544 24473 leader_election.cc:302] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:45.018949 24699 tablet_bootstrap.cc:654] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.019605 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.021168 24476 leader_election.cc:302] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e
I20240501 13:55:45.023977 24720 leader_election.cc:288] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.025477 24720 raft_consensus.cc:2781] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.025943 24720 raft_consensus.cc:482] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.026381 24720 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.027101 24719 raft_consensus.cc:2781] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.027823 24719 raft_consensus.cc:686] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.029073 24719 consensus_queue.cc:237] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.031472 24707 leader_election.cc:288] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
W20240501 13:55:45.038928 24476 leader_election.cc:341] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.042107 24699 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:45.042624 24699 ts_tablet_manager.cc:1367] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.004s
I20240501 13:55:45.043002 24720 raft_consensus.cc:504] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.044941 24705 raft_consensus.cc:482] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.045743 24700 tablet_bootstrap.cc:492] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.045585 24705 raft_consensus.cc:504] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.046247 24700 ts_tablet_manager.cc:1367] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.004s
I20240501 13:55:45.046967 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
W20240501 13:55:45.048305 24476 leader_election.cc:341] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.049225 24720 leader_election.cc:288] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.049311 24705 leader_election.cc:288] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.050483 24699 raft_consensus.cc:348] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.051182 24699 raft_consensus.cc:374] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.051493 24699 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.049026 24700 raft_consensus.cc:348] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.052210 24700 raft_consensus.cc:374] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.052579 24700 raft_consensus.cc:729] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.053154 24700 consensus_queue.cc:260] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.055166 24700 ts_tablet_manager.cc:1398] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240501 13:55:45.050364 24406 catalog_manager.cc:5484] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.056140 24700 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.056408 24701 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.056876 24701 ts_tablet_manager.cc:1367] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.021s
I20240501 13:55:45.059631 24701 raft_consensus.cc:348] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.060395 24701 raft_consensus.cc:374] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.060688 24701 raft_consensus.cc:729] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.060636 24699 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.061499 24701 consensus_queue.cc:260] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.063237 24699 ts_tablet_manager.cc:1398] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20240501 13:55:45.063947 24701 ts_tablet_manager.cc:1398] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 13:55:45.064179 24699 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:45.068985 24700 tablet_bootstrap.cc:654] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.073437 24701 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:45.074342 24699 tablet_bootstrap.cc:654] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.078114 24720 raft_consensus.cc:482] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.078752 24705 raft_consensus.cc:482] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.078833 24720 raft_consensus.cc:504] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.079391 24705 raft_consensus.cc:504] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.081421 24705 leader_election.cc:288] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.082577 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.083374 24659 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.083678 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
W20240501 13:55:45.085520 24476 leader_election.cc:341] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.087322 24720 leader_election.cc:288] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.088953 24699 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:45.089596 24699 ts_tablet_manager.cc:1367] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.006s
I20240501 13:55:45.092586 24659 raft_consensus.cc:2454] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 1.
I20240501 13:55:45.093540 24699 raft_consensus.cc:348] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.094334 24699 raft_consensus.cc:374] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.094643 24699 raft_consensus.cc:729] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.094821 24473 leader_election.cc:302] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.095525 24699 consensus_queue.cc:260] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.096550 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.098121 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.098857 24509 raft_consensus.cc:2454] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 0.
I20240501 13:55:45.099982 24701 tablet_bootstrap.cc:654] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.099669 24720 raft_consensus.cc:2781] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.100793 24720 raft_consensus.cc:686] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.101682 24720 consensus_queue.cc:237] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.102650 24700 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.103142 24700 ts_tablet_manager.cc:1367] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.004s
W20240501 13:55:45.105224 24625 leader_election.cc:341] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.098381 24699 ts_tablet_manager.cc:1398] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240501 13:55:45.108402 24625 leader_election.cc:302] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:45.109334 24705 raft_consensus.cc:2781] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.110050 24705 raft_consensus.cc:482] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.110484 24699 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:45.110560 24705 raft_consensus.cc:3037] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.109469 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.115237 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.115460 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.116614 24585 raft_consensus.cc:2454] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 0.
I20240501 13:55:45.121449 24705 raft_consensus.cc:504] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
W20240501 13:55:45.122372 24456 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:55:45.123109 24659 raft_consensus.cc:2379] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbaf3058782245828d49c901c40f1399 in current term 1: Already voted for candidate 2ad4fb9054d644748b36aac4798f724e in this term.
W20240501 13:55:45.123499 24625 leader_election.cc:341] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.123980 24705 leader_election.cc:288] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.125478 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.125159 24625 leader_election.cc:302] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.126101 24509 raft_consensus.cc:3037] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.126071 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.126711 24719 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.127158 24719 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.128666 24700 raft_consensus.cc:348] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.129192 24719 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.131305 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.131672 24701 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.132143 24701 ts_tablet_manager.cc:1367] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.010s
W20240501 13:55:45.132691 24476 leader_election.cc:341] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.134756 24701 raft_consensus.cc:348] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.135293 24701 raft_consensus.cc:374] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.135515 24701 raft_consensus.cc:729] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.136106 24701 consensus_queue.cc:260] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.138288 24701 ts_tablet_manager.cc:1398] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240501 13:55:45.138161 24473 leader_election.cc:302] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbaf3058782245828d49c901c40f1399; no voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.139266 24701 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
W20240501 13:55:45.140486 24625 leader_election.cc:341] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.140544 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.141628 24705 raft_consensus.cc:2781] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.130473 24700 raft_consensus.cc:374] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.142458 24659 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 0.
I20240501 13:55:45.142796 24700 raft_consensus.cc:729] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.144026 24473 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.145299 24723 raft_consensus.cc:2781] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.145696 24723 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.146266 24699 tablet_bootstrap.cc:654] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.146040 24723 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.145995 24700 consensus_queue.cc:260] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.149744 24700 ts_tablet_manager.cc:1398] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.046s user 0.010s sys 0.000s
I20240501 13:55:45.151199 24700 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.159054 24701 tablet_bootstrap.cc:654] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.155289 24723 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.161492 24509 raft_consensus.cc:2454] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 1.
I20240501 13:55:45.166229 24719 raft_consensus.cc:2726] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.170521 24625 leader_election.cc:302] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.177284 24700 tablet_bootstrap.cc:654] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.142608 24705 raft_consensus.cc:482] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.190070 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.189802 24406 catalog_manager.cc:5484] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } }
W20240501 13:55:45.191713 24476 leader_election.cc:341] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.193876 24735 raft_consensus.cc:482] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.194577 24735 raft_consensus.cc:504] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.197309 24735 leader_election.cc:288] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.197959 24735 raft_consensus.cc:2781] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.198475 24735 raft_consensus.cc:686] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.203126 24723 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.203472 24701 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.204002 24701 ts_tablet_manager.cc:1367] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.065s user 0.014s sys 0.018s
I20240501 13:55:45.182134 24705 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.210836 24699 tablet_bootstrap.cc:492] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:45.211480 24699 ts_tablet_manager.cc:1367] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.009s
I20240501 13:55:45.214913 24699 raft_consensus.cc:348] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.217229 24699 raft_consensus.cc:374] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.217522 24699 raft_consensus.cc:729] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.218187 24699 consensus_queue.cc:260] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.219960 24701 raft_consensus.cc:348] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.220749 24701 raft_consensus.cc:374] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.221710 24706 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.222115 24701 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.222208 24706 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
W20240501 13:55:45.222566 24625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.129:38341, user_credentials={real_user=slave}} blocked reactor thread for 53071.9us
I20240501 13:55:45.226022 24739 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.226357 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.226521 24739 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.227092 24659 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.228724 24739 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.230041 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.230443 24706 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.230736 24584 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 0.
I20240501 13:55:45.225286 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.199522 24735 consensus_queue.cc:237] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.231570 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.245198 24705 raft_consensus.cc:504] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.248952 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.249650 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.250586 24585 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 0.
W20240501 13:55:45.251224 24625 leader_election.cc:341] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:55:45.252190 24625 leader_election.cc:341] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.252930 24705 leader_election.cc:288] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.232096 24701 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.256071 24700 tablet_bootstrap.cc:492] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.256618 24700 ts_tablet_manager.cc:1367] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.005s
I20240501 13:55:45.256940 24701 ts_tablet_manager.cc:1398] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20240501 13:55:45.242286 24509 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 0.
I20240501 13:55:45.260306 24625 leader_election.cc:302] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:45.262923 24699 ts_tablet_manager.cc:1398] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.051s user 0.010s sys 0.001s
I20240501 13:55:45.268175 24706 raft_consensus.cc:482] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.269068 24699 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:45.268868 24706 raft_consensus.cc:504] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.271718 24706 leader_election.cc:288] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.261236 24705 raft_consensus.cc:2781] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.275463 24700 raft_consensus.cc:348] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.273609 24705 raft_consensus.cc:482] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.276259 24700 raft_consensus.cc:374] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.267588 24659 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 1.
I20240501 13:55:45.277312 24705 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.279361 24701 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:45.280058 24699 tablet_bootstrap.cc:654] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.280300 24473 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.281296 24723 raft_consensus.cc:2781] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.281741 24723 raft_consensus.cc:686] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.277429 24700 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.284742 24723 consensus_queue.cc:237] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.291522 24625 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.294298 24743 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.295011 24743 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.295346 24743 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.301347 24701 tablet_bootstrap.cc:654] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.301059 24700 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.305510 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.306321 24585 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.299548 24705 raft_consensus.cc:504] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.306746 24743 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.334612 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.335619 24659 raft_consensus.cc:2379] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in current term 1: Already voted for candidate 2ad4fb9054d644748b36aac4798f724e in this term.
I20240501 13:55:45.336156 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.336786 24658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.337262 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.338272 24584 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.335485 24743 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
W20240501 13:55:45.341035 24625 leader_election.cc:341] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.337652 24658 raft_consensus.cc:2454] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 0.
I20240501 13:55:45.346966 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.348611 24705 leader_election.cc:288] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:45.341801 24700 ts_tablet_manager.cc:1398] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.085s user 0.007s sys 0.000s
I20240501 13:55:45.349647 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.352089 24509 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.352787 24700 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.353354 24585 raft_consensus.cc:2454] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 1.
I20240501 13:55:45.354525 24625 leader_election.cc:302] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.355670 24705 raft_consensus.cc:2781] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.356191 24705 raft_consensus.cc:686] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.356812 24583 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.357371 24705 consensus_queue.cc:237] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.359129 24700 tablet_bootstrap.cc:654] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.362668 24548 leader_election.cc:302] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.363936 24706 raft_consensus.cc:2781] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.364459 24706 raft_consensus.cc:482] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.364881 24706 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.389253 24706 raft_consensus.cc:504] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.391065 24584 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 1.
I20240501 13:55:45.394578 24699 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:45.394937 24699 ts_tablet_manager.cc:1367] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.126s user 0.032s sys 0.028s
I20240501 13:55:45.387682 24509 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 1.
I20240501 13:55:45.395995 24701 tablet_bootstrap.cc:492] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.397089 24701 ts_tablet_manager.cc:1367] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.119s user 0.018s sys 0.012s
I20240501 13:55:45.384505 24405 catalog_manager.cc:5484] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 1 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:45.397272 24625 leader_election.cc:302] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:45.399155 24706 leader_election.cc:288] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.399544 24625 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.400241 24743 raft_consensus.cc:2781] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.398533 24583 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 1.
I20240501 13:55:45.403157 24723 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.403637 24723 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.400818 24743 raft_consensus.cc:686] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.405112 24743 consensus_queue.cc:237] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.406479 24699 raft_consensus.cc:348] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.407155 24699 raft_consensus.cc:374] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.407151 24723 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.407564 24699 raft_consensus.cc:729] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.407783 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.408254 24699 consensus_queue.cc:260] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
W20240501 13:55:45.409070 24476 leader_election.cc:341] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.410459 24408 catalog_manager.cc:5484] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.413110 24699 ts_tablet_manager.cc:1398] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240501 13:55:45.413309 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.414023 24699 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: Bootstrap starting.
I20240501 13:55:45.401890 24705 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.402215 24700 tablet_bootstrap.cc:492] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.414263 24659 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 0.
I20240501 13:55:45.414719 24700 ts_tablet_manager.cc:1367] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.000s
I20240501 13:55:45.414913 24705 raft_consensus.cc:686] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.415969 24705 consensus_queue.cc:237] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.416314 24473 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.417569 24700 raft_consensus.cc:348] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.418257 24700 raft_consensus.cc:374] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.417354 24701 raft_consensus.cc:348] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.418545 24700 raft_consensus.cc:729] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.402093 24658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.419030 24701 raft_consensus.cc:374] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.419135 24658 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.419489 24701 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.420310 24701 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.420953 24719 consensus_queue.cc:1035] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.422837 24701 ts_tablet_manager.cc:1398] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.025s user 0.001s sys 0.005s
I20240501 13:55:45.420306 24723 raft_consensus.cc:2781] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.426120 24723 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.426597 24723 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.430150 24700 consensus_queue.cc:260] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.431742 24701 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:45.437790 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.438531 24509 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 0.
I20240501 13:55:45.438567 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.439842 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.440268 24551 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e
I20240501 13:55:45.441443 24701 tablet_bootstrap.cc:654] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
W20240501 13:55:45.441501 24551 leader_election.cc:341] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:55:45.442471 24551 leader_election.cc:341] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:55:45.442499 24706 raft_consensus.cc:2741] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 13:55:45.426525 24658 raft_consensus.cc:2454] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 1.
I20240501 13:55:45.443032 24659 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.444020 24548 leader_election.cc:302] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.445619 24706 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.438082 24699 tablet_bootstrap.cc:654] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.446853 24706 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.450453 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.451861 24658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.452729 24658 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 0.
I20240501 13:55:45.454443 24548 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.455682 24706 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.456498 24706 raft_consensus.cc:2781] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.457172 24706 raft_consensus.cc:686] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:45.432658 24700 ts_tablet_manager.cc:1398] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240501 13:55:45.458736 24700 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.466476 24700 tablet_bootstrap.cc:654] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.434764 24723 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.469424 24719 consensus_queue.cc:1035] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.470464 24723 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.472669 24658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.473332 24658 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.472713 24510 raft_consensus.cc:2379] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in current term 1: Already voted for candidate bbaf3058782245828d49c901c40f1399 in this term.
I20240501 13:55:45.476315 24406 catalog_manager.cc:5484] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 1 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:45.478562 24406 catalog_manager.cc:5484] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 1 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:45.480437 24406 catalog_manager.cc:5484] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 1 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:45.466935 24699 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: No bootstrap required, opened a new log
I20240501 13:55:45.467572 24706 consensus_queue.cc:237] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.483729 24658 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 1.
I20240501 13:55:45.489972 24699 ts_tablet_manager.cc:1367] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.012s
I20240501 13:55:45.489751 24473 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:45.491058 24723 raft_consensus.cc:2781] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.491652 24723 raft_consensus.cc:686] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.492782 24723 consensus_queue.cc:237] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.497625 24753 raft_consensus.cc:2781] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.498078 24753 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.500383 24700 tablet_bootstrap.cc:492] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.507700 24700 ts_tablet_manager.cc:1367] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.015s
I20240501 13:55:45.506337 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.502105 24699 raft_consensus.cc:348] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.509702 24699 raft_consensus.cc:374] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.510042 24699 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.510716 24699 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.511744 24700 raft_consensus.cc:348] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.512944 24700 raft_consensus.cc:374] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.511785 24408 catalog_manager.cc:5484] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.513449 24700 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.518293 24700 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.498425 24753 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.520589 24699 ts_tablet_manager.cc:1398] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: Time spent starting tablet: real 0.030s user 0.006s sys 0.002s
I20240501 13:55:45.520779 24700 ts_tablet_manager.cc:1398] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
W20240501 13:55:45.521875 24476 consensus_peers.cc:485] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 -> Peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Couldn't send request to peer d7cc6e2dd96142d0993aacf2b5d1a6c7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240501 13:55:45.523016 24548 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.131:33637, user_credentials={real_user=slave}} blocked reactor thread for 68698.9us
I20240501 13:55:45.528003 24700 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.539539 24753 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.542533 24584 raft_consensus.cc:2379] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bbaf3058782245828d49c901c40f1399 in current term 1: Already voted for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in this term.
I20240501 13:55:45.544260 24701 tablet_bootstrap.cc:492] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.544924 24701 ts_tablet_manager.cc:1367] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.113s user 0.028s sys 0.022s
I20240501 13:55:45.548017 24701 raft_consensus.cc:348] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.548748 24701 raft_consensus.cc:374] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.549027 24701 raft_consensus.cc:729] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.549541 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.549706 24701 consensus_queue.cc:260] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.551878 24701 ts_tablet_manager.cc:1398] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:55:45.552824 24753 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.552953 24701 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: Bootstrap starting.
I20240501 13:55:45.553006 24706 raft_consensus.cc:482] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.553550 24706 raft_consensus.cc:504] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.554482 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.555263 24659 raft_consensus.cc:2379] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in current term 1: Already voted for candidate bbaf3058782245828d49c901c40f1399 in this term.
I20240501 13:55:45.555498 24706 leader_election.cc:288] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.556702 24700 tablet_bootstrap.cc:654] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.556977 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.557646 24510 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 0.
I20240501 13:55:45.558861 24701 tablet_bootstrap.cc:654] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.558738 24551 leader_election.cc:302] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.559646 24706 raft_consensus.cc:2781] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:55:45.560009 24706 raft_consensus.cc:482] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:55:45.560282 24706 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.562080 24700 tablet_bootstrap.cc:492] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.562615 24700 ts_tablet_manager.cc:1367] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.000s
I20240501 13:55:45.565304 24700 raft_consensus.cc:348] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.566161 24700 raft_consensus.cc:374] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.566480 24700 raft_consensus.cc:729] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.567232 24700 consensus_queue.cc:260] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.569633 24700 ts_tablet_manager.cc:1398] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 13:55:45.569700 24706 raft_consensus.cc:504] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.570693 24700 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Bootstrap starting.
I20240501 13:55:45.571825 24408 catalog_manager.cc:5484] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.573769 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.574666 24510 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.575799 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.576514 24659 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 0.
I20240501 13:55:45.577250 24700 tablet_bootstrap.cc:654] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Neither blocks nor log segments found. Creating new log.
I20240501 13:55:45.577975 24548 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.579159 24753 raft_consensus.cc:2726] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.583189 24706 leader_election.cc:288] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.585140 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.588063 24659 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.598785 24659 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 1.
I20240501 13:55:45.600453 24548 leader_election.cc:302] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.602413 24753 raft_consensus.cc:2781] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:55:45.603076 24753 raft_consensus.cc:686] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:45.604214 24753 consensus_queue.cc:237] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.609962 24510 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 1.
I20240501 13:55:45.623451 24701 tablet_bootstrap.cc:492] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: No bootstrap required, opened a new log
I20240501 13:55:45.623945 24701 ts_tablet_manager.cc:1367] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: Time spent bootstrapping tablet: real 0.071s user 0.030s sys 0.036s
I20240501 13:55:45.624222 24720 raft_consensus.cc:482] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.624828 24720 raft_consensus.cc:504] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.626433 24701 raft_consensus.cc:348] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.627346 24720 leader_election.cc:288] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.627988 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.626400 24406 catalog_manager.cc:5484] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 1 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.628923 24700 tablet_bootstrap.cc:492] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: No bootstrap required, opened a new log
I20240501 13:55:45.628974 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.629455 24700 ts_tablet_manager.cc:1367] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent bootstrapping tablet: real 0.059s user 0.025s sys 0.024s
I20240501 13:55:45.627629 24701 raft_consensus.cc:374] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.630066 24659 raft_consensus.cc:2379] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbaf3058782245828d49c901c40f1399 in current term 1: Already voted for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in this term.
I20240501 13:55:45.631291 24473 leader_election.cc:302] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbaf3058782245828d49c901c40f1399; no voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.630002 24701 raft_consensus.cc:729] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.632086 24700 raft_consensus.cc:348] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.632532 24720 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.632844 24700 raft_consensus.cc:374] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:55:45.633176 24700 raft_consensus.cc:729] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Initialized, Role: FOLLOWER
I20240501 13:55:45.632725 24701 consensus_queue.cc:260] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.633862 24700 consensus_queue.cc:260] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.635198 24701 ts_tablet_manager.cc:1398] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240501 13:55:45.636059 24700 ts_tablet_manager.cc:1398] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 13:55:45.638844 24720 raft_consensus.cc:2726] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.647405 24720 consensus_queue.cc:1035] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.674016 24753 raft_consensus.cc:482] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.674479 24753 raft_consensus.cc:504] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.675999 24753 leader_election.cc:288] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.676826 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.676999 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766fa8acf9e14ef2b47b2e9545d31dc4" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.677454 24510 raft_consensus.cc:2379] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in current term 1: Already voted for candidate 2ad4fb9054d644748b36aac4798f724e in this term.
I20240501 13:55:45.678256 24551 leader_election.cc:302] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.678901 24753 raft_consensus.cc:3037] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.683394 24753 raft_consensus.cc:2726] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.692261 24753 raft_consensus.cc:482] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.692790 24753 raft_consensus.cc:504] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:55:45.694316 24753 leader_election.cc:288] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637), bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341)
I20240501 13:55:45.695175 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.695345 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.696573 24551 leader_election.cc:302] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.697263 24753 raft_consensus.cc:2726] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.752862 24659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:50330
I20240501 13:55:45.753456 24659 raft_consensus.cc:595] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.757957 24723 raft_consensus.cc:482] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.758548 24723 raft_consensus.cc:504] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.760318 24723 leader_election.cc:288] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.761077 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" is_pre_election: true
I20240501 13:55:45.761735 24585 raft_consensus.cc:2379] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbaf3058782245828d49c901c40f1399 in current term 1: Already voted for candidate 2ad4fb9054d644748b36aac4798f724e in this term.
I20240501 13:55:45.761502 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.762905 24473 leader_election.cc:302] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbaf3058782245828d49c901c40f1399; no voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.763698 24723 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.765965 24753 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:55:45.766359 24753 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.767885 24753 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.769110 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ad4fb9054d644748b36aac4798f724e" is_pre_election: true
I20240501 13:55:45.769973 24659 raft_consensus.cc:2379] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in current term 1: Already voted for candidate bbaf3058782245828d49c901c40f1399 in this term.
I20240501 13:55:45.771356 24585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:39080
I20240501 13:55:45.771773 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbaf3058782245828d49c901c40f1399" is_pre_election: true
I20240501 13:55:45.771912 24585 raft_consensus.cc:595] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.773353 24551 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.774251 24753 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.773561 24723 raft_consensus.cc:2726] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.777953 24659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:50330
I20240501 13:55:45.778518 24659 raft_consensus.cc:595] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.780014 24753 raft_consensus.cc:2726] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:55:45.783224 24659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:50330
I20240501 13:55:45.783757 24659 raft_consensus.cc:595] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:45.791085 24705 consensus_queue.cc:1035] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.796833 24508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:34248
I20240501 13:55:45.797430 24508 raft_consensus.cc:595] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Received request to transfer leadership to TS d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.799245 24735 raft_consensus.cc:979] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.799741 24735 raft_consensus.cc:1067] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.804435 24705 consensus_queue.cc:1035] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.805676 24508 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34224
I20240501 13:55:45.806314 24508 raft_consensus.cc:482] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:45.806708 24508 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.807106 24510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:34248
I20240501 13:55:45.807828 24510 raft_consensus.cc:595] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Received request to transfer leadership to TS d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.813285 24508 raft_consensus.cc:504] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.816066 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.817868 24508 leader_election.cc:288] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.819075 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd70340327a488aa092093096d4bf1b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.819233 24508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:34248
I20240501 13:55:45.819744 24659 raft_consensus.cc:3032] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:45.819936 24508 raft_consensus.cc:595] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Received request to transfer leadership to TS d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.820137 24659 raft_consensus.cc:729] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.821074 24659 consensus_queue.cc:260] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.822101 24659 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.822908 24508 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.825605 24705 consensus_queue.cc:1035] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:55:45.829924 24659 raft_consensus.cc:2454] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:45.831060 24473 leader_election.cc:302] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters: d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:55:45.831804 24723 raft_consensus.cc:2781] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:45.832273 24723 raft_consensus.cc:686] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.832665 24705 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.833209 24723 consensus_queue.cc:237] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.841372 24743 consensus_queue.cc:1035] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.849099 24405 catalog_manager.cc:5484] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 1 to 2, leader changed from 2ad4fb9054d644748b36aac4798f724e (127.23.70.131) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.855348 24705 raft_consensus.cc:979] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.855854 24735 raft_consensus.cc:1067] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.858507 24509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34224
I20240501 13:55:45.860255 24509 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:45.860706 24509 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.862047 24735 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.868629 24723 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.871136 24723 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.873387 24509 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.879251 24735 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.886633 24509 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.887905 24585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.888453 24585 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.888906 24657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.889492 24657 raft_consensus.cc:3032] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:45.889798 24657 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.890565 24657 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.891490 24657 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.896092 24585 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:45.897434 24476 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.898355 24723 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:45.898864 24723 raft_consensus.cc:686] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.899721 24723 consensus_queue.cc:237] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.903475 24657 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:45.908970 24405 catalog_manager.cc:5484] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 1 to 2, leader changed from 2ad4fb9054d644748b36aac4798f724e (127.23.70.131) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.913443 24743 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.918978 24720 raft_consensus.cc:979] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: : Instructing follower d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:45.919441 24720 raft_consensus.cc:1067] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Signalling peer d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:45.922490 24583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:39058
I20240501 13:55:45.923075 24583 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:45.923540 24583 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.928805 24585 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:55:45.933672 24705 raft_consensus.cc:979] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.934156 24705 raft_consensus.cc:1067] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:45.937291 24509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34224
I20240501 13:55:45.937969 24509 raft_consensus.cc:482] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:45.938427 24509 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.940587 24583 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.941972 24584 raft_consensus.cc:1226] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Rejecting Update request from peer bbaf3058782245828d49c901c40f1399 for earlier term 1. Current term is 2. Ops: []
I20240501 13:55:45.943763 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:45.944425 24510 raft_consensus.cc:3032] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:45.944718 24510 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.945420 24510 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.946362 24510 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.946154 24509 raft_consensus.cc:504] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.948513 24509 leader_election.cc:288] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.949476 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:45.950106 24584 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.952971 24583 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:45.953069 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.953722 24659 raft_consensus.cc:3032] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:45.954071 24659 raft_consensus.cc:729] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:45.954977 24510 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:45.954941 24657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:45.954887 24659 consensus_queue.cc:260] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.955703 24657 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.956440 24659 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:45.958200 24584 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:45.959245 24476 leader_election.cc:302] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.960094 24707 raft_consensus.cc:2781] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:45.960575 24707 raft_consensus.cc:686] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:45.961392 24707 consensus_queue.cc:237] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.963227 24659 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:45.963176 24657 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:45.964655 24551 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:45.965430 24753 raft_consensus.cc:2781] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:45.965950 24753 raft_consensus.cc:686] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:45.966733 24753 consensus_queue.cc:237] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:45.971508 24406 catalog_manager.cc:5484] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 1 to 2, leader changed from 2ad4fb9054d644748b36aac4798f724e (127.23.70.131) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.974107 24408 catalog_manager.cc:5484] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 1 to 2, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 2 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:45.978247 24753 consensus_queue.cc:1035] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:45.990234 24706 consensus_queue.cc:1035] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:46.079695 24707 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:46.091502 24720 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:46.099181 24723 raft_consensus.cc:979] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: : Instructing follower d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:46.099696 24720 raft_consensus.cc:1067] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Signalling peer d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:46.101233 24584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:39058
I20240501 13:55:46.101872 24584 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:46.102250 24584 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.104259 24723 raft_consensus.cc:979] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: : Instructing follower d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:46.104679 24720 raft_consensus.cc:1067] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Signalling peer d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:55:46.106201 24583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:39058
I20240501 13:55:46.106765 24583 raft_consensus.cc:482] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:46.107146 24583 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.108232 24584 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.111109 24584 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:46.111229 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:46.111869 24510 raft_consensus.cc:3032] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:46.112179 24510 raft_consensus.cc:729] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:46.112562 24657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:46.113170 24657 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.112953 24510 consensus_queue.cc:260] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.113847 24583 raft_consensus.cc:504] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.114293 24510 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.115849 24583 leader_election.cc:288] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:46.116874 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:46.116874 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120f3f71269e4400962f36d05d1ae459" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:46.117624 24659 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.117699 24509 raft_consensus.cc:3032] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:46.118147 24509 raft_consensus.cc:729] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:46.118911 24509 consensus_queue.cc:260] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.120149 24509 raft_consensus.cc:3037] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.121371 24657 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:46.122375 24510 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:46.123056 24548 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:46.124269 24706 raft_consensus.cc:2781] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:46.124763 24706 raft_consensus.cc:686] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:46.125370 24706 consensus_queue.cc:237] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.128551 24509 raft_consensus.cc:2454] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:46.124279 24659 raft_consensus.cc:2454] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 2.
I20240501 13:55:46.130697 24551 leader_election.cc:302] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:46.131515 24756 raft_consensus.cc:2781] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:46.131976 24756 raft_consensus.cc:686] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:46.132815 24756 consensus_queue.cc:237] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.132756 24406 catalog_manager.cc:5484] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 1 to 2, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 2 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:46.145560 24406 catalog_manager.cc:5484] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 1 to 2, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 2 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:46.215422 24657 raft_consensus.cc:1261] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.216601 24720 consensus_queue.cc:1035] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.223278 24584 raft_consensus.cc:3037] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.227205 24706 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:46.230432 24584 raft_consensus.cc:1261] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.231912 24723 consensus_queue.cc:1035] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.233819 24706 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:55:46.236825 24753 raft_consensus.cc:979] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:46.237268 24753 raft_consensus.cc:1067] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:46.239797 24509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34238
I20240501 13:55:46.243777 24509 raft_consensus.cc:482] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:46.244187 24509 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.251142 24509 raft_consensus.cc:504] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.255960 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:46.256681 24583 raft_consensus.cc:3032] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:46.257011 24583 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:46.257764 24583 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.258343 24509 leader_election.cc:288] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:46.259292 24583 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.260488 24657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:46.261189 24657 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:46.264648 24583 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:46.265789 24476 leader_election.cc:302] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:46.266494 24723 raft_consensus.cc:2781] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:46.267024 24723 raft_consensus.cc:686] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:46.267007 24657 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:46.268102 24723 consensus_queue.cc:237] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:46.275157 24405 catalog_manager.cc:5484] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 1 to 2, leader changed from d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:46.311724 24583 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.313154 24723 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.320009 24657 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.321697 24707 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.367216 24657 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.368530 24756 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.376318 24509 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.377899 24756 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 13:55:46.523110 24657 raft_consensus.cc:1261] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.524259 24707 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.531452 24583 raft_consensus.cc:1261] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.533371 24723 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:55:46.558166 24657 raft_consensus.cc:1261] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.559296 24756 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.566104 24509 raft_consensus.cc:1261] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.567446 24756 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.715667 24509 raft_consensus.cc:1261] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.716729 24583 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.716912 24756 consensus_queue.cc:1035] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.718014 24707 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.728809 24659 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.731235 24657 raft_consensus.cc:1261] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:46.731084 24707 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:46.732596 24753 consensus_queue.cc:1035] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:48.855685 23834 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240501 13:55:48.856406 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:48.872959 24509 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:55:48.873528 24509 raft_consensus.cc:595] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Received request to transfer leadership to TS 2ad4fb9054d644748b36aac4798f724e
I20240501 13:55:48.876248 24509 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:55:48.876854 24509 raft_consensus.cc:595] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Received request to transfer leadership to TS 2ad4fb9054d644748b36aac4798f724e
I20240501 13:55:48.877951 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 13:55:48.878453 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:48.903934 24707 raft_consensus.cc:979] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: : Instructing follower 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:55:48.904423 24793 raft_consensus.cc:1067] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Signalling peer 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:55:48.905968 24659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:50322
I20240501 13:55:48.906462 24659 raft_consensus.cc:482] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:48.906723 24659 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:48.911468 24659 raft_consensus.cc:504] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:48.913208 24659 leader_election.cc:288] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:48.914321 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:48.914546 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:48.914861 24583 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:48.915107 24509 raft_consensus.cc:3032] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:48.915395 24509 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:48.916018 24509 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:48.916983 24509 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:48.919790 24583 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:55:48.920922 24625 leader_election.cc:302] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:48.921602 24735 raft_consensus.cc:2781] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:48.921695 24509 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:55:48.922997 24735 raft_consensus.cc:686] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:48.923884 24735 consensus_queue.cc:237] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:48.931039 24405 catalog_manager.cc:5484] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 2 to 3, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 3 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:49.115878 24707 raft_consensus.cc:979] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: : Instructing follower 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:55:49.116397 24707 raft_consensus.cc:1067] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Signalling peer 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:55:49.117877 24657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:50322
I20240501 13:55:49.118422 24657 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:49.118703 24657 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:49.123693 24657 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:49.125288 24657 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:55:49.126309 24509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:55:49.126453 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:49.126814 24509 raft_consensus.cc:3032] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:49.126988 24583 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:49.127343 24509 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:49.128064 24509 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:49.129093 24509 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:49.131826 24583 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:55:49.132786 24625 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:49.133456 24735 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:49.133831 24735 raft_consensus.cc:686] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:49.133854 24509 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:55:49.134651 24735 consensus_queue.cc:237] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:49.142064 24405 catalog_manager.cc:5484] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 2 to 3, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 3 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:55:49.357460 24583 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:49.358711 24735 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:49.368273 24509 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:49.370076 24735 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:49.613377 24509 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:49.614753 24735 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:49.623996 24583 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:49.628070 24795 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:49.879390 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:49.885262 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:49.885659 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:50.886584 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:50.892910 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:50.893433 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:51.894616 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:51.901980 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:51.902465 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:52.903275 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:52.909601 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:52.910014 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:53.910797 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:53.915843 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:53.916260 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:54.917029 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:54.922569 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:54.922995 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:55.923859 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:55.929044 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:55.929463 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:56.930277 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:56.936478 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:56.936851 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:57.937646 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:55:57.942807 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:55:57.943226 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:55:58.970197 23834 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240501 13:55:58.977048 24659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:50330
I20240501 13:55:58.977830 24659 raft_consensus.cc:595] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:58.982591 24659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:50330
I20240501 13:55:58.983093 24659 raft_consensus.cc:595] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:58.992300 24583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:39080
I20240501 13:55:58.992879 24583 raft_consensus.cc:595] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:58.997936 24583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:39080
I20240501 13:55:58.998418 24583 raft_consensus.cc:595] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:59.002897 24583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb834bb158944475a2efb3c109076eed"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
mode: GRACEFUL
new_leader_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:39080
I20240501 13:55:59.003399 24583 raft_consensus.cc:595] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Received request to transfer leadership to TS bbaf3058782245828d49c901c40f1399
I20240501 13:55:59.094208 24821 raft_consensus.cc:979] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.094767 24819 raft_consensus.cc:1067] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.096365 24510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34238
I20240501 13:55:59.096875 24510 raft_consensus.cc:482] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:59.097216 24510 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.103787 24510 raft_consensus.cc:504] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.105577 24510 leader_election.cc:288] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 3 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:59.106576 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:59.106576 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9992986258f4becb6995d28b5c3127d" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:59.107522 24659 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.107592 24583 raft_consensus.cc:3032] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:59.108026 24583 raft_consensus.cc:729] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:59.108860 24583 consensus_queue.cc:260] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.110287 24583 raft_consensus.cc:3037] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.114840 24659 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 3.
I20240501 13:55:59.116189 24473 leader_election.cc:302] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:59.116988 24820 raft_consensus.cc:2781] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:59.117204 24583 raft_consensus.cc:2454] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 3.
I20240501 13:55:59.117358 24820 raft_consensus.cc:686] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:59.118245 24820 consensus_queue.cc:237] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.125134 24823 raft_consensus.cc:979] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.125674 24817 raft_consensus.cc:1067] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.125514 24408 catalog_manager.cc:5484] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 2 to 3, leader changed from d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 3 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:59.127122 24510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34224
I20240501 13:55:59.127694 24510 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:59.128131 24510 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.134265 24510 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.137485 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:59.138330 24584 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.138660 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:59.139107 24659 raft_consensus.cc:3032] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:55:59.139329 24659 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:59.139935 24659 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.143085 24659 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.147996 24510 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 4 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:59.150265 24584 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 4.
I20240501 13:55:59.151140 24659 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 4.
I20240501 13:55:59.151427 24476 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:59.152240 24820 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:55:59.152650 24820 raft_consensus.cc:686] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:59.153295 24820 consensus_queue.cc:237] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.160005 24408 catalog_manager.cc:5484] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 3 to 4, leader changed from 2ad4fb9054d644748b36aac4798f724e (127.23.70.131) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 4 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:59.283912 24821 raft_consensus.cc:979] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.284360 24819 raft_consensus.cc:1067] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.285741 24510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34238
I20240501 13:55:59.286243 24510 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:59.286512 24510 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.292182 24510 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.293812 24510 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 3 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:59.294747 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:59.294854 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:59.295238 24659 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.295397 24584 raft_consensus.cc:3032] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:55:59.295747 24584 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:59.296491 24584 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.297542 24584 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:55:59.300370 24659 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 3.
I20240501 13:55:59.301465 24473 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:59.302112 24820 raft_consensus.cc:2781] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:55:59.302214 24584 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 3.
I20240501 13:55:59.302942 24820 raft_consensus.cc:686] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:59.304526 24820 consensus_queue.cc:237] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.312172 24408 catalog_manager.cc:5484] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 2 to 3, leader changed from d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 3 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:59.348512 24823 raft_consensus.cc:979] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.349045 24817 raft_consensus.cc:1067] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.350611 24510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34224
I20240501 13:55:59.351094 24510 raft_consensus.cc:482] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:59.351363 24510 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.356225 24510 raft_consensus.cc:504] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.357779 24510 leader_election.cc:288] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 4 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:59.358788 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:59.358772 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99207ce180c4f8b91e00bfbe9e9bc6b" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:59.359495 24659 raft_consensus.cc:3032] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:55:59.359522 24584 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.359966 24659 raft_consensus.cc:729] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:55:59.360761 24659 consensus_queue.cc:260] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.361936 24659 raft_consensus.cc:3037] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:55:59.366288 24584 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 4.
I20240501 13:55:59.367955 24476 leader_election.cc:302] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbaf3058782245828d49c901c40f1399, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:55:59.368121 24659 raft_consensus.cc:2454] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 4.
I20240501 13:55:59.368728 24820 raft_consensus.cc:2781] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:55:59.369184 24820 raft_consensus.cc:686] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:59.369920 24820 consensus_queue.cc:237] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.376811 24408 catalog_manager.cc:5484] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 3 to 4, leader changed from 2ad4fb9054d644748b36aac4798f724e (127.23.70.131) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 4 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:59.424983 24821 raft_consensus.cc:979] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: : Instructing follower bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.425390 24819 raft_consensus.cc:1067] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Signalling peer bbaf3058782245828d49c901c40f1399 to start an election
I20240501 13:55:59.426664 24510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb834bb158944475a2efb3c109076eed"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
from {username='slave'} at 127.0.0.1:34238
I20240501 13:55:59.427102 24510 raft_consensus.cc:482] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:55:59.427359 24510 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:59.431816 24510 raft_consensus.cc:504] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.433808 24510 leader_election.cc:288] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:55:59.434722 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:55:59.434722 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb834bb158944475a2efb3c109076eed" candidate_uuid: "bbaf3058782245828d49c901c40f1399" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:55:59.435468 24659 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:59.435523 24584 raft_consensus.cc:3032] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:55:59.435922 24584 raft_consensus.cc:729] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:55:59.436681 24584 consensus_queue.cc:260] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.437884 24584 raft_consensus.cc:3037] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:55:59.441756 24659 raft_consensus.cc:2454] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:59.442772 24473 leader_election.cc:302] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, bbaf3058782245828d49c901c40f1399; no voters:
I20240501 13:55:59.443403 24820 raft_consensus.cc:2781] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:55:59.443475 24584 raft_consensus.cc:2454] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbaf3058782245828d49c901c40f1399 in term 2.
I20240501 13:55:59.443811 24820 raft_consensus.cc:686] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Leader. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:55:59.444710 24820 consensus_queue.cc:237] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:55:59.450949 24408 catalog_manager.cc:5484] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 reported cstate change: term changed from 1 to 2, leader changed from d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130) to bbaf3058782245828d49c901c40f1399 (127.23.70.129). New cstate: current_term: 2 leader_uuid: "bbaf3058782245828d49c901c40f1399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:55:59.573590 24584 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:59.574812 24820 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:59.588783 24659 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:59.590909 24812 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:55:59.662695 24659 raft_consensus.cc:1261] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:59.663794 24812 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:59.670087 24584 raft_consensus.cc:1261] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:59.671944 24820 consensus_queue.cc:1035] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:59.833315 24583 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:59.834656 24834 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:59.844892 24659 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:55:59.846210 24834 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:55:59.853139 24657 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:59.854349 24834 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:55:59.867467 24583 raft_consensus.cc:1261] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:55:59.869082 24812 consensus_queue.cc:1035] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:55:59.976660 24659 raft_consensus.cc:1261] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:59.977782 24812 consensus_queue.cc:1035] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:55:59.983444 24583 raft_consensus.cc:1261] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer bbaf3058782245828d49c901c40f1399: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:55:59.984812 24820 consensus_queue.cc:1035] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:02.037361 23834 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240501 13:56:02.038308 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:02.045730 24510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "045ac8899eb64675a94d3955f9816670"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:56:02.046478 24510 raft_consensus.cc:595] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Received request to transfer leadership to TS 2ad4fb9054d644748b36aac4798f724e
I20240501 13:56:02.049120 24510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:56:02.049650 24510 raft_consensus.cc:595] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Received request to transfer leadership to TS d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:56:02.052000 24510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:56:02.052529 24510 raft_consensus.cc:595] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Received request to transfer leadership to TS d7cc6e2dd96142d0993aacf2b5d1a6c7
I20240501 13:56:02.054605 24510 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6"
dest_uuid: "bbaf3058782245828d49c901c40f1399"
mode: GRACEFUL
new_leader_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:59214
I20240501 13:56:02.055074 24510 raft_consensus.cc:595] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Received request to transfer leadership to TS 2ad4fb9054d644748b36aac4798f724e
I20240501 13:56:02.056015 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 13:56:02.056515 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:02.242465 24834 raft_consensus.cc:979] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: : Instructing follower d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:56:02.242894 24834 raft_consensus.cc:1067] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Signalling peer d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:56:02.244338 24583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:39058
I20240501 13:56:02.244803 24583 raft_consensus.cc:482] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:56:02.245124 24583 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:56:02.250756 24583 raft_consensus.cc:504] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.252735 24583 leader_election.cc:288] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 4 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:56:02.253894 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:56:02.254446 24659 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:56:02.254530 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5350250ac144378a3cabb34c92412a" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:56:02.255064 24510 raft_consensus.cc:3032] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 13:56:02.255321 24510 raft_consensus.cc:729] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:56:02.256067 24510 consensus_queue.cc:260] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.257130 24510 raft_consensus.cc:3037] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Advancing to term 4
I20240501 13:56:02.259795 24659 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 4.
I20240501 13:56:02.261049 24548 leader_election.cc:302] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:56:02.262003 24819 raft_consensus.cc:2781] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 13:56:02.262888 24510 raft_consensus.cc:2454] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 4.
I20240501 13:56:02.263062 24819 raft_consensus.cc:686] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:56:02.264187 24819 consensus_queue.cc:237] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.269310 24850 raft_consensus.cc:979] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: : Instructing follower 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:56:02.269815 24834 raft_consensus.cc:1067] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Signalling peer 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:56:02.271281 24659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:50322
I20240501 13:56:02.271703 24659 raft_consensus.cc:482] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:56:02.272051 24659 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:56:02.272478 24408 catalog_manager.cc:5484] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 3 to 4, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 4 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:02.277530 24659 raft_consensus.cc:504] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.279665 24659 leader_election.cc:288] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689)
I20240501 13:56:02.280694 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:56:02.280543 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5f0c2a27344cb0af1f5b4f8adbb4f6" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:56:02.281226 24583 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:56:02.281332 24510 raft_consensus.cc:3032] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 13:56:02.281769 24510 raft_consensus.cc:729] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:56:02.282804 24510 consensus_queue.cc:260] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.284564 24510 raft_consensus.cc:3037] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Advancing to term 3
I20240501 13:56:02.287204 24583 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:56:02.288478 24625 leader_election.cc:302] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:56:02.289371 24849 raft_consensus.cc:2781] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:56:02.291306 24849 raft_consensus.cc:686] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:56:02.292150 24510 raft_consensus.cc:2454] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 3.
I20240501 13:56:02.292522 24849 consensus_queue.cc:237] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.299827 24408 catalog_manager.cc:5484] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 2 to 3, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 3 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:02.347401 24850 raft_consensus.cc:979] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: : Instructing follower 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:56:02.347842 24834 raft_consensus.cc:1067] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Signalling peer 2ad4fb9054d644748b36aac4798f724e to start an election
I20240501 13:56:02.349277 24659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "045ac8899eb64675a94d3955f9816670"
dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
from {username='slave'} at 127.0.0.1:50322
I20240501 13:56:02.349737 24659 raft_consensus.cc:482] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:56:02.350041 24659 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:02.354399 24659 raft_consensus.cc:504] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:56:02.355901 24659 leader_election.cc:288] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 2 election: Requested vote from peers d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689), bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341)
I20240501 13:56:02.357177 24583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
I20240501 13:56:02.357280 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "045ac8899eb64675a94d3955f9816670" candidate_uuid: "2ad4fb9054d644748b36aac4798f724e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:56:02.357856 24583 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:02.357995 24510 raft_consensus.cc:3032] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:56:02.358395 24510 raft_consensus.cc:729] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:56:02.359184 24510 consensus_queue.cc:260] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:56:02.360167 24510 raft_consensus.cc:3037] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:02.362722 24583 raft_consensus.cc:2454] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 2.
I20240501 13:56:02.363691 24625 leader_election.cc:302] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:56:02.364552 24849 raft_consensus.cc:2781] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:56:02.365024 24849 raft_consensus.cc:686] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 2 LEADER]: Becoming Leader. State: Replica: 2ad4fb9054d644748b36aac4798f724e, State: Running, Role: LEADER
I20240501 13:56:02.365726 24510 raft_consensus.cc:2454] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ad4fb9054d644748b36aac4798f724e in term 2.
I20240501 13:56:02.365928 24849 consensus_queue.cc:237] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } }
I20240501 13:56:02.373234 24408 catalog_manager.cc:5484] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e reported cstate change: term changed from 1 to 2, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to 2ad4fb9054d644748b36aac4798f724e (127.23.70.131). New cstate: current_term: 2 leader_uuid: "2ad4fb9054d644748b36aac4798f724e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:02.421854 24850 raft_consensus.cc:979] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: : Instructing follower d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:56:02.422313 24834 raft_consensus.cc:1067] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Signalling peer d7cc6e2dd96142d0993aacf2b5d1a6c7 to start an election
I20240501 13:56:02.423523 24583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c83855d66734897831036f82ae28e94"
dest_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7"
from {username='slave'} at 127.0.0.1:39058
I20240501 13:56:02.423952 24583 raft_consensus.cc:482] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:56:02.424218 24583 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Advancing to term 5
I20240501 13:56:02.428388 24583 raft_consensus.cc:504] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.429883 24583 leader_election.cc:288] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 5 election: Requested vote from peers bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341), 2ad4fb9054d644748b36aac4798f724e (127.23.70.131:33637)
I20240501 13:56:02.430752 24510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "bbaf3058782245828d49c901c40f1399"
I20240501 13:56:02.430907 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c83855d66734897831036f82ae28e94" candidate_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2ad4fb9054d644748b36aac4798f724e"
I20240501 13:56:02.431257 24510 raft_consensus.cc:3032] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 13:56:02.431354 24659 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Advancing to term 5
I20240501 13:56:02.431529 24510 raft_consensus.cc:729] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bbaf3058782245828d49c901c40f1399, State: Running, Role: LEADER
I20240501 13:56:02.432138 24510 consensus_queue.cc:260] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.433053 24510 raft_consensus.cc:3037] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Advancing to term 5
I20240501 13:56:02.435937 24659 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 5.
I20240501 13:56:02.436928 24548 leader_election.cc:302] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad4fb9054d644748b36aac4798f724e, d7cc6e2dd96142d0993aacf2b5d1a6c7; no voters:
I20240501 13:56:02.437394 24510 raft_consensus.cc:2454] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc6e2dd96142d0993aacf2b5d1a6c7 in term 5.
I20240501 13:56:02.437794 24819 raft_consensus.cc:2781] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 13:56:02.438251 24819 raft_consensus.cc:686] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 5 LEADER]: Becoming Leader. State: Replica: d7cc6e2dd96142d0993aacf2b5d1a6c7, State: Running, Role: LEADER
I20240501 13:56:02.439055 24819 consensus_queue.cc:237] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } }
I20240501 13:56:02.445322 24408 catalog_manager.cc:5484] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 reported cstate change: term changed from 4 to 5, leader changed from bbaf3058782245828d49c901c40f1399 (127.23.70.129) to d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130). New cstate: current_term: 5 leader_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:02.667146 24659 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:56:02.668304 24819 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:02.678220 24510 raft_consensus.cc:1261] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 13:56:02.679770 24819 consensus_queue.cc:1035] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:02.747792 24510 raft_consensus.cc:1261] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:56:02.748899 24849 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:02.757804 24583 raft_consensus.cc:1261] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 13:56:02.759286 24857 consensus_queue.cc:1035] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:02.761024 24510 raft_consensus.cc:1261] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:56:02.764040 24857 consensus_queue.cc:1035] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:02.773393 24583 raft_consensus.cc:1261] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Refusing update from remote peer 2ad4fb9054d644748b36aac4798f724e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:56:02.776396 24867 consensus_queue.cc:1035] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc6e2dd96142d0993aacf2b5d1a6c7" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:02.896670 24659 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 5 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 13:56:02.897688 24819 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad4fb9054d644748b36aac4798f724e" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 33637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 13:56:02.906119 24510 raft_consensus.cc:1261] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 5 FOLLOWER]: Refusing update from remote peer d7cc6e2dd96142d0993aacf2b5d1a6c7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 13:56:02.907794 24872 consensus_queue.cc:1035] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbaf3058782245828d49c901c40f1399" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 38341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 13:56:03.057452 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:03.063010 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:03.063481 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:04.064277 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:04.069675 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:04.070183 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:05.071030 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:05.076308 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:05.076731 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:06.077626 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:06.083902 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:06.084370 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:07.085196 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:07.091342 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:07.091773 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:08.092665 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:08.099269 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:08.099704 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:09.100507 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:09.106065 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:09.106524 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:10.107409 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:10.113979 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:10.114409 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:11.115222 23834 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:56:11.122467 23834 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:56:11.122860 23834 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:56:12.150359 23834 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240501 13:56:12.151198 23834 tablet_server.cc:178] TabletServer@127.23.70.129:0 shutting down...
W20240501 13:56:12.179466 24551 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20240501 13:56:12.182163 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:56:12.183023 23834 tablet_replica.cc:330] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399: stopping tablet replica
W20240501 13:56:12.183224 24551 consensus_peers.cc:485] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 -> Peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Couldn't send request to peer bbaf3058782245828d49c901c40f1399. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:12.183914 23834 raft_consensus.cc:2229] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.184949 23834 raft_consensus.cc:2258] T cb834bb158944475a2efb3c109076eed P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.186882 23834 tablet_replica.cc:330] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.187448 23834 raft_consensus.cc:2229] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.188313 23834 raft_consensus.cc:2258] T f9992986258f4becb6995d28b5c3127d P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.190027 23834 tablet_replica.cc:330] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.190515 23834 raft_consensus.cc:2229] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.190917 23834 raft_consensus.cc:2258] T 766fa8acf9e14ef2b47b2e9545d31dc4 P bbaf3058782245828d49c901c40f1399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.192945 23834 tablet_replica.cc:330] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.193445 23834 raft_consensus.cc:2229] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 4 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.194211 23834 raft_consensus.cc:2258] T c99207ce180c4f8b91e00bfbe9e9bc6b P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.195981 23834 tablet_replica.cc:330] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.196504 23834 raft_consensus.cc:2229] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.196977 23834 raft_consensus.cc:2258] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P bbaf3058782245828d49c901c40f1399 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.198691 23834 tablet_replica.cc:330] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.199123 23834 raft_consensus.cc:2229] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.199483 23834 raft_consensus.cc:2258] T 045ac8899eb64675a94d3955f9816670 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.201220 23834 tablet_replica.cc:330] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.201699 23834 raft_consensus.cc:2229] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.202100 23834 raft_consensus.cc:2258] T ab5350250ac144378a3cabb34c92412a P bbaf3058782245828d49c901c40f1399 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.203804 23834 tablet_replica.cc:330] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.204288 23834 raft_consensus.cc:2229] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.205109 23834 raft_consensus.cc:2258] T ccd70340327a488aa092093096d4bf1b P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.206797 23834 tablet_replica.cc:330] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.207249 23834 raft_consensus.cc:2229] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.207646 23834 raft_consensus.cc:2258] T 8c83855d66734897831036f82ae28e94 P bbaf3058782245828d49c901c40f1399 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.209364 23834 tablet_replica.cc:330] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399: stopping tablet replica
I20240501 13:56:12.209870 23834 raft_consensus.cc:2229] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.210328 23834 raft_consensus.cc:2258] T 120f3f71269e4400962f36d05d1ae459 P bbaf3058782245828d49c901c40f1399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.241374 23834 tablet_server.cc:194] TabletServer@127.23.70.129:0 shutdown complete.
I20240501 13:56:12.259685 23834 tablet_server.cc:178] TabletServer@127.23.70.130:0 shutting down...
W20240501 13:56:12.269806 24551 consensus_peers.cc:485] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 -> Peer bbaf3058782245828d49c901c40f1399 (127.23.70.129:38341): Couldn't send request to peer bbaf3058782245828d49c901c40f1399. Status: Network error: Client connection negotiation failed: client connection to 127.23.70.129:38341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:12.290169 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:56:12.290902 23834 tablet_replica.cc:330] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.291975 23834 raft_consensus.cc:2229] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.292630 23834 raft_consensus.cc:2258] T 766fa8acf9e14ef2b47b2e9545d31dc4 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.294595 23834 tablet_replica.cc:330] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.295221 23834 raft_consensus.cc:2229] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.295635 23834 raft_consensus.cc:2258] T f9992986258f4becb6995d28b5c3127d P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.297363 23834 tablet_replica.cc:330] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.297803 23834 raft_consensus.cc:2229] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.298177 23834 raft_consensus.cc:2258] T 045ac8899eb64675a94d3955f9816670 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.299822 23834 tablet_replica.cc:330] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.300304 23834 raft_consensus.cc:2229] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.301124 23834 raft_consensus.cc:2258] T ab5350250ac144378a3cabb34c92412a P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.302966 23834 tablet_replica.cc:330] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.303464 23834 raft_consensus.cc:2229] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.303853 23834 raft_consensus.cc:2258] T cb834bb158944475a2efb3c109076eed P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.305471 23834 tablet_replica.cc:330] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.305934 23834 raft_consensus.cc:2229] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 5 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.306701 23834 raft_consensus.cc:2258] T 8c83855d66734897831036f82ae28e94 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.308537 23834 tablet_replica.cc:330] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.308972 23834 raft_consensus.cc:2229] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.309352 23834 raft_consensus.cc:2258] T ccd70340327a488aa092093096d4bf1b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.311151 23834 tablet_replica.cc:330] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.311623 23834 raft_consensus.cc:2229] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.312515 23834 raft_consensus.cc:2258] T 120f3f71269e4400962f36d05d1ae459 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.314368 23834 tablet_replica.cc:330] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.314832 23834 raft_consensus.cc:2229] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.315266 23834 raft_consensus.cc:2258] T c99207ce180c4f8b91e00bfbe9e9bc6b P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.316962 23834 tablet_replica.cc:330] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7: stopping tablet replica
I20240501 13:56:12.317384 23834 raft_consensus.cc:2229] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.317786 23834 raft_consensus.cc:2258] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P d7cc6e2dd96142d0993aacf2b5d1a6c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.346715 23834 tablet_server.cc:194] TabletServer@127.23.70.130:0 shutdown complete.
W20240501 13:56:12.363977 24625 consensus_peers.cc:485] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e -> Peer d7cc6e2dd96142d0993aacf2b5d1a6c7 (127.23.70.130:33689): Couldn't send request to peer d7cc6e2dd96142d0993aacf2b5d1a6c7. Status: Network error: Client connection negotiation failed: client connection to 127.23.70.130:33689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:12.368921 23834 tablet_server.cc:178] TabletServer@127.23.70.131:0 shutting down...
I20240501 13:56:12.397792 23834 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:56:12.398504 23834 tablet_replica.cc:330] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.399273 23834 raft_consensus.cc:2229] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.399782 23834 raft_consensus.cc:2258] T f9992986258f4becb6995d28b5c3127d P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.401707 23834 tablet_replica.cc:330] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.402241 23834 raft_consensus.cc:2229] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.402664 23834 raft_consensus.cc:2258] T cb834bb158944475a2efb3c109076eed P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.404461 23834 tablet_replica.cc:330] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.404946 23834 raft_consensus.cc:2229] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.405447 23834 raft_consensus.cc:2258] T ab5350250ac144378a3cabb34c92412a P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.407266 23834 tablet_replica.cc:330] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.407815 23834 raft_consensus.cc:2229] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.408721 23834 raft_consensus.cc:2258] T 045ac8899eb64675a94d3955f9816670 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.410763 23834 tablet_replica.cc:330] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.411278 23834 raft_consensus.cc:2229] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.412070 23834 raft_consensus.cc:2258] T 766fa8acf9e14ef2b47b2e9545d31dc4 P 2ad4fb9054d644748b36aac4798f724e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.413928 23834 tablet_replica.cc:330] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.414403 23834 raft_consensus.cc:2229] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.414800 23834 raft_consensus.cc:2258] T 120f3f71269e4400962f36d05d1ae459 P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.416337 23834 tablet_replica.cc:330] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.416846 23834 raft_consensus.cc:2229] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.417280 23834 raft_consensus.cc:2258] T ccd70340327a488aa092093096d4bf1b P 2ad4fb9054d644748b36aac4798f724e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.418958 23834 tablet_replica.cc:330] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.419425 23834 raft_consensus.cc:2229] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.419832 23834 raft_consensus.cc:2258] T 8c83855d66734897831036f82ae28e94 P 2ad4fb9054d644748b36aac4798f724e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.421588 23834 tablet_replica.cc:330] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.422039 23834 raft_consensus.cc:2229] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:12.422435 23834 raft_consensus.cc:2258] T c99207ce180c4f8b91e00bfbe9e9bc6b P 2ad4fb9054d644748b36aac4798f724e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.424039 23834 tablet_replica.cc:330] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e: stopping tablet replica
I20240501 13:56:12.424548 23834 raft_consensus.cc:2229] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.425311 23834 raft_consensus.cc:2258] T fe5f0c2a27344cb0af1f5b4f8adbb4f6 P 2ad4fb9054d644748b36aac4798f724e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.453553 23834 tablet_server.cc:194] TabletServer@127.23.70.131:0 shutdown complete.
I20240501 13:56:12.473198 23834 master.cc:537] Master@127.23.70.190:44121 shutting down...
I20240501 13:56:12.498651 23834 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:56:12.499226 23834 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:12.499552 23834 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9a0c08db05244b4bdc9d4787d7acea7: stopping tablet replica
I20240501 13:56:12.517555 23834 master.cc:552] Master@127.23.70.190:44121 shutdown complete.
2ad4fb9054d644748b36aac4798f724e 1
bbaf3058782245828d49c901c40f1399 5
d7cc6e2dd96142d0993aacf2b5d1a6c7 4
2ad4fb9054d644748b36aac4798f724e 3
bbaf3058782245828d49c901c40f1399 3
d7cc6e2dd96142d0993aacf2b5d1a6c7 4
2ad4fb9054d644748b36aac4798f724e 1
bbaf3058782245828d49c901c40f1399 8
d7cc6e2dd96142d0993aacf2b5d1a6c7 1
2ad4fb9054d644748b36aac4798f724e 3
bbaf3058782245828d49c901c40f1399 4
d7cc6e2dd96142d0993aacf2b5d1a6c7 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28598 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240501 13:56:12.562057 23834 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.23.70.190:46015
I20240501 13:56:12.563170 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:56:12.570003 24891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:12.571264 24892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:12.572371 24894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:12.573282 23834 server_base.cc:1034] running on GCE node
I20240501 13:56:12.574276 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:56:12.574512 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:56:12.574671 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571772574655 us; error 0 us; skew 500 ppm
I20240501 13:56:12.575237 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:56:12.584118 23834 webserver.cc:460] Webserver started at http://127.23.70.190:37059/ using document root <none> and password file <none>
I20240501 13:56:12.584697 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:56:12.584892 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:56:12.585211 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:56:12.586508 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/master-0-root/instance:
uuid: "c418f0193ebd43ffa6e23d2a5520d985"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.591267 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 13:56:12.594653 24899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.595464 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 13:56:12.595752 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/master-0-root
uuid: "c418f0193ebd43ffa6e23d2a5520d985"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.596038 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:56:12.623770 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:56:12.625118 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:56:12.667832 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.190:46015
I20240501 13:56:12.667922 24950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.190:46015 every 8 connection(s)
I20240501 13:56:12.671892 24951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:12.684293 24951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985: Bootstrap starting.
I20240501 13:56:12.689051 24951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:12.693418 24951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985: No bootstrap required, opened a new log
I20240501 13:56:12.695495 24951 raft_consensus.cc:348] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER }
I20240501 13:56:12.696013 24951 raft_consensus.cc:374] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:12.696209 24951 raft_consensus.cc:729] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c418f0193ebd43ffa6e23d2a5520d985, State: Initialized, Role: FOLLOWER
I20240501 13:56:12.696831 24951 consensus_queue.cc:260] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER }
I20240501 13:56:12.697353 24951 raft_consensus.cc:388] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:56:12.697597 24951 raft_consensus.cc:482] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:56:12.697923 24951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:12.702062 24951 raft_consensus.cc:504] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER }
I20240501 13:56:12.702594 24951 leader_election.cc:302] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c418f0193ebd43ffa6e23d2a5520d985; no voters:
I20240501 13:56:12.703688 24951 leader_election.cc:288] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:56:12.704100 24954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:12.705579 24954 raft_consensus.cc:686] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [term 1 LEADER]: Becoming Leader. State: Replica: c418f0193ebd43ffa6e23d2a5520d985, State: Running, Role: LEADER
I20240501 13:56:12.706224 24954 consensus_queue.cc:237] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER }
I20240501 13:56:12.706686 24951 sys_catalog.cc:564] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:56:12.711716 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c418f0193ebd43ffa6e23d2a5520d985. Latest consensus state: current_term: 1 leader_uuid: "c418f0193ebd43ffa6e23d2a5520d985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER } }
I20240501 13:56:12.711849 24955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c418f0193ebd43ffa6e23d2a5520d985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c418f0193ebd43ffa6e23d2a5520d985" member_type: VOTER } }
I20240501 13:56:12.712905 24955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [sys.catalog]: This master's current role is: LEADER
I20240501 13:56:12.714361 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985 [sys.catalog]: This master's current role is: LEADER
I20240501 13:56:12.715479 24962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:56:12.723968 24962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:56:12.724992 23834 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:56:12.733053 24962 catalog_manager.cc:1347] Generated new cluster ID: de420ca149fe49789b4a7c435cecc462
I20240501 13:56:12.733381 24962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:56:12.759974 24962 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:56:12.761265 24962 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:56:12.775794 24962 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c418f0193ebd43ffa6e23d2a5520d985: Generated new TSK 0
I20240501 13:56:12.776456 24962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:56:12.791481 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:56:12.798507 24972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:12.799891 24973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:12.801894 23834 server_base.cc:1034] running on GCE node
W20240501 13:56:12.802549 24975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:12.803503 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:56:12.803723 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:56:12.803881 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571772803863 us; error 0 us; skew 500 ppm
I20240501 13:56:12.804508 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:56:12.807158 23834 webserver.cc:460] Webserver started at http://127.23.70.129:41761/ using document root <none> and password file <none>
I20240501 13:56:12.807699 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:56:12.807895 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:56:12.808169 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:56:12.809561 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/instance:
uuid: "346b2bb141744d459f1fa3ad602d1a15"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.814570 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 13:56:12.818035 24980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.818823 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 13:56:12.819126 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root
uuid: "346b2bb141744d459f1fa3ad602d1a15"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.819437 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:56:12.848290 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:56:12.849609 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:56:12.851501 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:56:12.854282 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:56:12.854569 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.854848 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:56:12.855069 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.899029 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.129:33309
I20240501 13:56:12.899129 25042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.129:33309 every 8 connection(s)
I20240501 13:56:12.903889 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:56:12.913122 25048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:12.916234 25047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:12.919498 25043 heartbeater.cc:347] Connected to a master server at 127.23.70.190:46015
I20240501 13:56:12.919896 25043 heartbeater.cc:464] Registering TS with master...
W20240501 13:56:12.920027 25050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:12.920836 25043 heartbeater.cc:511] Master 127.23.70.190:46015 requested a full tablet report, sending...
I20240501 13:56:12.921531 23834 server_base.cc:1034] running on GCE node
I20240501 13:56:12.923020 24916 ts_manager.cc:194] Registered new tserver with Master: 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309)
I20240501 13:56:12.923041 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:56:12.923399 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:56:12.923568 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571772923549 us; error 0 us; skew 500 ppm
I20240501 13:56:12.924125 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:56:12.925091 24916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33918
I20240501 13:56:12.926899 23834 webserver.cc:460] Webserver started at http://127.23.70.130:33779/ using document root <none> and password file <none>
I20240501 13:56:12.927414 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:56:12.927603 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:56:12.927852 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:56:12.929040 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/instance:
uuid: "171c5cc5f6df4820bc16946f8717737e"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.933701 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 13:56:12.936971 25055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.937812 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 13:56:12.938071 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root
uuid: "171c5cc5f6df4820bc16946f8717737e"
format_stamp: "Formatted at 2024-05-01 13:56:12 on dist-test-slave-s72h"
I20240501 13:56:12.938325 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:56:12.974109 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:56:12.975423 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:56:12.976866 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:56:12.979107 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:56:12.979317 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:12.979564 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:56:12.979720 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:13.020255 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.130:34553
I20240501 13:56:13.020447 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.130:34553 every 8 connection(s)
I20240501 13:56:13.024937 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:56:13.032311 25121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:13.035373 25122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:13.036166 25124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:13.036590 25118 heartbeater.cc:347] Connected to a master server at 127.23.70.190:46015
I20240501 13:56:13.036923 25118 heartbeater.cc:464] Registering TS with master...
I20240501 13:56:13.037607 25118 heartbeater.cc:511] Master 127.23.70.190:46015 requested a full tablet report, sending...
I20240501 13:56:13.038094 23834 server_base.cc:1034] running on GCE node
I20240501 13:56:13.038959 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:56:13.039206 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:56:13.039434 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571773039412 us; error 0 us; skew 500 ppm
I20240501 13:56:13.039701 24916 ts_manager.cc:194] Registered new tserver with Master: 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:13.040138 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:56:13.041349 24916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33934
I20240501 13:56:13.042917 23834 webserver.cc:460] Webserver started at http://127.23.70.131:41575/ using document root <none> and password file <none>
I20240501 13:56:13.043596 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:56:13.043864 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:56:13.044133 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:56:13.045307 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/instance:
uuid: "ef020bad8d24499281f69de00e90af06"
format_stamp: "Formatted at 2024-05-01 13:56:13 on dist-test-slave-s72h"
I20240501 13:56:13.049818 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 13:56:13.053232 25129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:13.054131 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:56:13.054461 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root
uuid: "ef020bad8d24499281f69de00e90af06"
format_stamp: "Formatted at 2024-05-01 13:56:13 on dist-test-slave-s72h"
I20240501 13:56:13.054733 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:56:13.067401 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:56:13.068625 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:56:13.070137 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:56:13.072494 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:56:13.072693 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:13.072952 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:56:13.073094 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:13.114207 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.131:40317
I20240501 13:56:13.114307 25191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.131:40317 every 8 connection(s)
I20240501 13:56:13.127696 25192 heartbeater.cc:347] Connected to a master server at 127.23.70.190:46015
I20240501 13:56:13.128051 25192 heartbeater.cc:464] Registering TS with master...
I20240501 13:56:13.128787 25192 heartbeater.cc:511] Master 127.23.70.190:46015 requested a full tablet report, sending...
I20240501 13:56:13.130653 24916 ts_manager.cc:194] Registered new tserver with Master: ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:13.131088 23834 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01354366s
I20240501 13:56:13.131984 24916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33940
I20240501 13:56:13.133525 23834 test_util.cc:268] Using random seed: -646266179
I20240501 13:56:13.159077 24916 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33956:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 13:56:13.165197 24916 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:56:13.379828 25008 tablet_service.cc:1467] Processing CreateTablet for tablet c77fc3f2c34f4b58bb2af4583495adba (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 13:56:13.381283 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77fc3f2c34f4b58bb2af4583495adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.382758 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 13:56:13.384133 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.384999 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 8a816d7705ea4eed878efa007b669731 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 13:56:13.386365 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a816d7705ea4eed878efa007b669731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.386591 25007 tablet_service.cc:1467] Processing CreateTablet for tablet 3add9205fa474b098334f36bc4cb3eef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 13:56:13.387857 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add9205fa474b098334f36bc4cb3eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.389529 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 67a090c154b9467b8e2cad3029c44cee (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 13:56:13.390872 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a090c154b9467b8e2cad3029c44cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.393533 25005 tablet_service.cc:1467] Processing CreateTablet for tablet 56eceec85d2f42f5a0baf57bb03b45f4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 13:56:13.398486 25004 tablet_service.cc:1467] Processing CreateTablet for tablet f5858563bd674a6a8ed86d4b6b174cb2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 13:56:13.400820 25003 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 13:56:13.401607 25001 tablet_service.cc:1467] Processing CreateTablet for tablet d46a9c65a336412d97c25350438ffba9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 13:56:13.405443 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 3add9205fa474b098334f36bc4cb3eef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 13:56:13.405670 25081 tablet_service.cc:1467] Processing CreateTablet for tablet 67a090c154b9467b8e2cad3029c44cee (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 13:56:13.407001 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add9205fa474b098334f36bc4cb3eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.407135 25079 tablet_service.cc:1467] Processing CreateTablet for tablet f5858563bd674a6a8ed86d4b6b174cb2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 13:56:13.407646 25083 tablet_service.cc:1467] Processing CreateTablet for tablet c77fc3f2c34f4b58bb2af4583495adba (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 13:56:13.404007 25002 tablet_service.cc:1467] Processing CreateTablet for tablet e58e0e1a34b84e78b395e8776f8a48ed (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 13:56:13.408746 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56eceec85d2f42f5a0baf57bb03b45f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.405537 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 56eceec85d2f42f5a0baf57bb03b45f4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 13:56:13.409507 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5858563bd674a6a8ed86d4b6b174cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.409260 25078 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 13:56:13.410612 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58e0e1a34b84e78b395e8776f8a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.411301 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.411226 25077 tablet_service.cc:1467] Processing CreateTablet for tablet e58e0e1a34b84e78b395e8776f8a48ed (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 13:56:13.413089 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a9c65a336412d97c25350438ffba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.413442 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77fc3f2c34f4b58bb2af4583495adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.415947 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5858563bd674a6a8ed86d4b6b174cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.418254 25076 tablet_service.cc:1467] Processing CreateTablet for tablet d46a9c65a336412d97c25350438ffba9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 13:56:13.421528 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56eceec85d2f42f5a0baf57bb03b45f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.423130 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.424669 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a090c154b9467b8e2cad3029c44cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.426311 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a9c65a336412d97c25350438ffba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.427791 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58e0e1a34b84e78b395e8776f8a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.439891 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 8a816d7705ea4eed878efa007b669731 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 13:56:13.444782 25207 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:13.444917 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a816d7705ea4eed878efa007b669731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.449947 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 13:56:13.451472 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.460917 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba50ad350b4473cb475aa22f2ab76ef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 13:56:13.461465 25156 tablet_service.cc:1467] Processing CreateTablet for tablet 3add9205fa474b098334f36bc4cb3eef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 13:56:13.462296 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba50ad350b4473cb475aa22f2ab76ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.462803 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3add9205fa474b098334f36bc4cb3eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.463754 24999 tablet_service.cc:1467] Processing CreateTablet for tablet a3e4ec10c6ef41a9815b71da267be33f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 13:56:13.465500 25155 tablet_service.cc:1467] Processing CreateTablet for tablet 67a090c154b9467b8e2cad3029c44cee (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 13:56:13.467594 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a090c154b9467b8e2cad3029c44cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.469506 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 13:56:13.470889 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d8be04ae2de44a7afa4abbcc4ec3bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.473862 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e4ec10c6ef41a9815b71da267be33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.473537 25151 tablet_service.cc:1467] Processing CreateTablet for tablet e58e0e1a34b84e78b395e8776f8a48ed (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 13:56:13.467187 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 56eceec85d2f42f5a0baf57bb03b45f4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 13:56:13.486747 25150 tablet_service.cc:1467] Processing CreateTablet for tablet d46a9c65a336412d97c25350438ffba9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 13:56:13.481060 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58e0e1a34b84e78b395e8776f8a48ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.492806 25207 tablet_bootstrap.cc:654] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:13.496807 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56eceec85d2f42f5a0baf57bb03b45f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.482213 25157 tablet_service.cc:1467] Processing CreateTablet for tablet c77fc3f2c34f4b58bb2af4583495adba (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 13:56:13.508981 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77fc3f2c34f4b58bb2af4583495adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.513839 25148 tablet_service.cc:1467] Processing CreateTablet for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 13:56:13.483729 25153 tablet_service.cc:1467] Processing CreateTablet for tablet f5858563bd674a6a8ed86d4b6b174cb2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 13:56:13.517364 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d46a9c65a336412d97c25350438ffba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.517135 25008 tablet_service.cc:1467] Processing CreateTablet for tablet 70b9d51a2e094f0d8b45171217807a23 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 13:56:13.519887 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b9d51a2e094f0d8b45171217807a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.527781 25003 tablet_service.cc:1467] Processing CreateTablet for tablet 55e60d3e56d348209c02fb051fd3d27f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 13:56:13.529253 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e60d3e56d348209c02fb051fd3d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.490001 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 8a816d7705ea4eed878efa007b669731 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 13:56:13.520951 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b65b2aa3bc84d22bf7ca8258043cc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.538671 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 3aae555c69934908afd09a93e6191d17 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 13:56:13.540148 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aae555c69934908afd09a93e6191d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.552510 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 0b683a5a430741e6ac1949303d905584 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 13:56:13.554369 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b683a5a430741e6ac1949303d905584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.569078 25007 tablet_service.cc:1467] Processing CreateTablet for tablet f5fbdfb5256246c5b6701c5f49c3deb9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 13:56:13.571542 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fbdfb5256246c5b6701c5f49c3deb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.574637 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 13:56:13.576120 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.578994 25003 tablet_service.cc:1467] Processing CreateTablet for tablet bebe230e116f4ecd84b6b251fa299925 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 13:56:13.580399 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe230e116f4ecd84b6b251fa299925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.595198 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 73740a9c3d7443cea7a555b0367a2b66 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 13:56:13.599396 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73740a9c3d7443cea7a555b0367a2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.601476 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5858563bd674a6a8ed86d4b6b174cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.514592 25002 tablet_service.cc:1467] Processing CreateTablet for tablet c110d748768044dabb2ba88e75ce883b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 13:56:13.610508 25078 tablet_service.cc:1467] Processing CreateTablet for tablet a3e4ec10c6ef41a9815b71da267be33f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 13:56:13.629333 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e4ec10c6ef41a9815b71da267be33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.611032 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a816d7705ea4eed878efa007b669731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.610702 25076 tablet_service.cc:1467] Processing CreateTablet for tablet 70b9d51a2e094f0d8b45171217807a23 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 13:56:13.612381 25081 tablet_service.cc:1467] Processing CreateTablet for tablet c110d748768044dabb2ba88e75ce883b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 13:56:13.612696 25077 tablet_service.cc:1467] Processing CreateTablet for tablet 3aae555c69934908afd09a93e6191d17 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 13:56:13.610508 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba50ad350b4473cb475aa22f2ab76ef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 13:56:13.615970 25001 tablet_service.cc:1467] Processing CreateTablet for tablet 357df45144824ef386e687e270aa8e77 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 13:56:13.593895 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7e26b87b054270bc41f6e57ff69ac0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 13:56:13.634804 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aae555c69934908afd09a93e6191d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.640666 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c110d748768044dabb2ba88e75ce883b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.642215 25207 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:13.649075 25207 ts_tablet_manager.cc:1367] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.204s user 0.018s sys 0.019s
I20240501 13:56:13.650491 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba50ad350b4473cb475aa22f2ab76ef (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 13:56:13.651947 25207 raft_consensus.cc:348] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:13.652724 25207 raft_consensus.cc:374] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:13.652913 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c110d748768044dabb2ba88e75ce883b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.653023 25207 raft_consensus.cc:729] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:13.653887 25207 consensus_queue.cc:260] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:13.660120 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b9d51a2e094f0d8b45171217807a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.665608 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba50ad350b4473cb475aa22f2ab76ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.673532 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357df45144824ef386e687e270aa8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.678627 25210 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:13.680859 25209 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:13.681550 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7e26b87b054270bc41f6e57ff69ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.685096 25210 tablet_bootstrap.cc:654] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:13.698365 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 0b683a5a430741e6ac1949303d905584 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 13:56:13.699858 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b683a5a430741e6ac1949303d905584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.675829 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba50ad350b4473cb475aa22f2ab76ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.705229 25083 tablet_service.cc:1467] Processing CreateTablet for tablet f5fbdfb5256246c5b6701c5f49c3deb9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 13:56:13.706619 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fbdfb5256246c5b6701c5f49c3deb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.707046 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 13:56:13.710948 25080 tablet_service.cc:1467] Processing CreateTablet for tablet bebe230e116f4ecd84b6b251fa299925 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 13:56:13.715240 25005 tablet_service.cc:1467] Processing CreateTablet for tablet 530c7a7742ff4dc182758c2250c1fabd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 13:56:13.717167 25156 tablet_service.cc:1467] Processing CreateTablet for tablet a3e4ec10c6ef41a9815b71da267be33f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 13:56:13.718539 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e4ec10c6ef41a9815b71da267be33f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:56:13.719209 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:13.705231 25155 tablet_service.cc:1467] Processing CreateTablet for tablet 70b9d51a2e094f0d8b45171217807a23 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 13:56:13.720377 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70b9d51a2e094f0d8b45171217807a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.738000 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.740463 25007 tablet_service.cc:1467] Processing CreateTablet for tablet d1d28bb928464ca7b989d4597a5b1d9c (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 13:56:13.741881 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 8a49139fcaa8462593981602f13840c4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 13:56:13.699968 25004 tablet_service.cc:1467] Processing CreateTablet for tablet 8e985a77279a46ec98e6d3164f7756c2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 13:56:13.743315 25209 tablet_bootstrap.cc:654] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:13.748839 25151 tablet_service.cc:1467] Processing CreateTablet for tablet 3aae555c69934908afd09a93e6191d17 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 13:56:13.750237 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aae555c69934908afd09a93e6191d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.758816 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530c7a7742ff4dc182758c2250c1fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.765301 25207 ts_tablet_manager.cc:1398] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.116s user 0.007s sys 0.017s
I20240501 13:56:13.769279 25006 tablet_service.cc:1467] Processing CreateTablet for tablet c1954780552c46cb829f499d56d225c7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240501 13:56:13.770764 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1954780552c46cb829f499d56d225c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.771476 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 55e60d3e56d348209c02fb051fd3d27f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 13:56:13.772455 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe230e116f4ecd84b6b251fa299925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.772802 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e60d3e56d348209c02fb051fd3d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.780702 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a49139fcaa8462593981602f13840c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.780741 25043 heartbeater.cc:503] Master 127.23.70.190:46015 was elected leader, sending a full tablet report...
I20240501 13:56:13.780532 25150 tablet_service.cc:1467] Processing CreateTablet for tablet c110d748768044dabb2ba88e75ce883b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 13:56:13.782511 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c110d748768044dabb2ba88e75ce883b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.782440 25153 tablet_service.cc:1467] Processing CreateTablet for tablet f5fbdfb5256246c5b6701c5f49c3deb9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 13:56:13.784133 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5fbdfb5256246c5b6701c5f49c3deb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.799754 25209 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:13.800311 25209 ts_tablet_manager.cc:1367] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.002s
I20240501 13:56:13.803319 25209 raft_consensus.cc:348] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:13.805260 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 0b683a5a430741e6ac1949303d905584 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 13:56:13.805588 25002 tablet_service.cc:1467] Processing CreateTablet for tablet b14ca53436a04447b62816101d8b838b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 13:56:13.806180 25207 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:13.806558 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e985a77279a46ec98e6d3164f7756c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.807441 25076 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7e26b87b054270bc41f6e57ff69ac0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 13:56:13.807716 25008 tablet_service.cc:1467] Processing CreateTablet for tablet aa1fd409e4d541dca63ed7a0544b4c5e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 13:56:13.808995 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1fd409e4d541dca63ed7a0544b4c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.810688 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14ca53436a04447b62816101d8b838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.811434 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 13:56:13.804152 25209 raft_consensus.cc:374] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:13.812426 25209 raft_consensus.cc:729] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:13.724134 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 55e60d3e56d348209c02fb051fd3d27f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 13:56:13.815603 25210 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:13.816210 25210 ts_tablet_manager.cc:1367] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.138s user 0.015s sys 0.050s
I20240501 13:56:13.808995 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7e26b87b054270bc41f6e57ff69ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.813232 25209 consensus_queue.cc:260] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:13.825183 25001 tablet_service.cc:1467] Processing CreateTablet for tablet b7e50604381e4450b9f3c000fe9030fd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 13:56:13.826627 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e50604381e4450b9f3c000fe9030fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.837603 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d28bb928464ca7b989d4597a5b1d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.840982 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b683a5a430741e6ac1949303d905584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.840921 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55e60d3e56d348209c02fb051fd3d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.856853 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ec4e2e661b4705afa2e4a2c4a76bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.862843 25148 tablet_service.cc:1467] Processing CreateTablet for tablet bebe230e116f4ecd84b6b251fa299925 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 13:56:13.877838 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebe230e116f4ecd84b6b251fa299925. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.873776 25207 tablet_bootstrap.cc:654] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:13.830399 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 73740a9c3d7443cea7a555b0367a2b66 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 13:56:13.880013 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73740a9c3d7443cea7a555b0367a2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.830044 25077 tablet_service.cc:1467] Processing CreateTablet for tablet 8e985a77279a46ec98e6d3164f7756c2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 13:56:13.885701 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e985a77279a46ec98e6d3164f7756c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.885810 25210 raft_consensus.cc:348] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:13.886687 25210 raft_consensus.cc:374] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:13.886993 25210 raft_consensus.cc:729] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:13.887759 25210 consensus_queue.cc:260] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:13.941185 25083 tablet_service.cc:1467] Processing CreateTablet for tablet d1d28bb928464ca7b989d4597a5b1d9c (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 13:56:13.942658 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d28bb928464ca7b989d4597a5b1d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.944607 25192 heartbeater.cc:503] Master 127.23.70.190:46015 was elected leader, sending a full tablet report...
I20240501 13:56:13.935146 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 357df45144824ef386e687e270aa8e77 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 13:56:13.946502 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357df45144824ef386e687e270aa8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:13.955979 25211 raft_consensus.cc:482] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:13.956579 25211 raft_consensus.cc:504] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:13.963671 25210 ts_tablet_manager.cc:1398] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.147s user 0.008s sys 0.012s
I20240501 13:56:13.964339 25211 leader_election.cc:288] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:13.979897 25118 heartbeater.cc:503] Master 127.23.70.190:46015 was elected leader, sending a full tablet report...
I20240501 13:56:13.980790 25216 raft_consensus.cc:482] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:13.981458 25216 raft_consensus.cc:504] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:13.986714 25216 leader_election.cc:288] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:14.009066 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 17458082ba194fc8b534019c9eab0485 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 13:56:14.010354 25151 tablet_service.cc:1467] Processing CreateTablet for tablet 73740a9c3d7443cea7a555b0367a2b66 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 13:56:14.011405 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73740a9c3d7443cea7a555b0367a2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.036511 25155 tablet_service.cc:1467] Processing CreateTablet for tablet d1d28bb928464ca7b989d4597a5b1d9c (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 13:56:14.037837 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d28bb928464ca7b989d4597a5b1d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.043170 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 821bad94d33240cd9802ff777bcb842a (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 13:56:14.044473 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821bad94d33240cd9802ff777bcb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.048971 25005 tablet_service.cc:1467] Processing CreateTablet for tablet 72f1ef588c2a42b4b724543c340ab182 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 13:56:14.050428 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f1ef588c2a42b4b724543c340ab182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.053159 25156 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7e26b87b054270bc41f6e57ff69ac0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 13:56:14.054549 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7e26b87b054270bc41f6e57ff69ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.057298 25210 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:14.062227 25006 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae8ea0d9e240c29296a5b826f67196 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 13:56:14.065124 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae8ea0d9e240c29296a5b826f67196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.066041 25003 tablet_service.cc:1467] Processing CreateTablet for tablet e8dff22b19a440d3bf74e5b47c061727 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 13:56:14.013698 25150 tablet_service.cc:1467] Processing CreateTablet for tablet 357df45144824ef386e687e270aa8e77 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 13:56:14.081457 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 357df45144824ef386e687e270aa8e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.082185 25207 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:14.082854 25207 ts_tablet_manager.cc:1367] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.277s user 0.031s sys 0.002s
I20240501 13:56:14.082538 25006 tablet_service.cc:1467] Processing CreateTablet for tablet bdc140167ab54483b3db40ba7a61937f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 13:56:14.084573 25209 ts_tablet_manager.cc:1398] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.284s user 0.017s sys 0.044s
I20240501 13:56:14.085597 25209 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:14.086583 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 4e912ea2c2074a20b15c82376861b3f7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 13:56:14.066138 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 8e985a77279a46ec98e6d3164f7756c2 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 13:56:14.093034 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e985a77279a46ec98e6d3164f7756c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.106230 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:14.114030 24982 leader_election.cc:341] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.118906 25078 tablet_service.cc:1467] Processing CreateTablet for tablet 8a49139fcaa8462593981602f13840c4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 13:56:14.139429 25004 tablet_service.cc:1467] Processing CreateTablet for tablet e059d8fd99574a5eac0b49e9405d7bd4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 13:56:14.068202 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dff22b19a440d3bf74e5b47c061727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.171242 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17458082ba194fc8b534019c9eab0485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.177103 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e912ea2c2074a20b15c82376861b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.180815 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e059d8fd99574a5eac0b49e9405d7bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.182093 25151 tablet_service.cc:1467] Processing CreateTablet for tablet c1954780552c46cb829f499d56d225c7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240501 13:56:14.169793 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 8a49139fcaa8462593981602f13840c4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 13:56:14.161136 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a49139fcaa8462593981602f13840c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.184073 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1954780552c46cb829f499d56d225c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.192449 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 530c7a7742ff4dc182758c2250c1fabd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 13:56:14.199934 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530c7a7742ff4dc182758c2250c1fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.202507 25207 raft_consensus.cc:348] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.203441 25207 raft_consensus.cc:374] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.203804 25207 raft_consensus.cc:729] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.204771 25207 consensus_queue.cc:260] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.207399 25207 ts_tablet_manager.cc:1398] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.124s user 0.010s sys 0.001s
I20240501 13:56:14.208492 25207 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:14.203776 25075 tablet_service.cc:1467] Processing CreateTablet for tablet aa1fd409e4d541dca63ed7a0544b4c5e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 13:56:14.209784 25153 tablet_service.cc:1467] Processing CreateTablet for tablet b14ca53436a04447b62816101d8b838b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 13:56:14.211076 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14ca53436a04447b62816101d8b838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.212674 25079 tablet_service.cc:1467] Processing CreateTablet for tablet b14ca53436a04447b62816101d8b838b (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 13:56:14.193787 25080 tablet_service.cc:1467] Processing CreateTablet for tablet b7e50604381e4450b9f3c000fe9030fd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 13:56:14.116786 25081 tablet_service.cc:1467] Processing CreateTablet for tablet c1954780552c46cb829f499d56d225c7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240501 13:56:14.220882 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14ca53436a04447b62816101d8b838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.221985 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e50604381e4450b9f3c000fe9030fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.221647 25148 tablet_service.cc:1467] Processing CreateTablet for tablet aa1fd409e4d541dca63ed7a0544b4c5e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 13:56:14.223740 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1fd409e4d541dca63ed7a0544b4c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.224650 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1954780552c46cb829f499d56d225c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.228634 25153 tablet_service.cc:1467] Processing CreateTablet for tablet b7e50604381e4450b9f3c000fe9030fd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 13:56:14.230013 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e50604381e4450b9f3c000fe9030fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.236424 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc140167ab54483b3db40ba7a61937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.244304 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 72f1ef588c2a42b4b724543c340ab182 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 13:56:14.245719 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f1ef588c2a42b4b724543c340ab182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.247555 25153 tablet_service.cc:1467] Processing CreateTablet for tablet 17458082ba194fc8b534019c9eab0485 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 13:56:14.248914 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17458082ba194fc8b534019c9eab0485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.250898 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 72f1ef588c2a42b4b724543c340ab182 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 13:56:14.253082 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f1ef588c2a42b4b724543c340ab182. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.255072 25008 tablet_service.cc:1467] Processing CreateTablet for tablet 94e26289285d4f14b97aa17f829a7c96 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240501 13:56:14.265884 25214 raft_consensus.cc:482] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:14.272514 25157 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae8ea0d9e240c29296a5b826f67196 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 13:56:14.190788 25001 tablet_service.cc:1467] Processing CreateTablet for tablet 021330a06d7d4faf9350427cbf308ed9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 13:56:14.289211 25214 raft_consensus.cc:504] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.292035 25214 leader_election.cc:288] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:14.277151 25153 tablet_service.cc:1467] Processing CreateTablet for tablet 821bad94d33240cd9802ff777bcb842a (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 13:56:14.224246 25209 tablet_bootstrap.cc:654] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.284632 25155 tablet_service.cc:1467] Processing CreateTablet for tablet bdc140167ab54483b3db40ba7a61937f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 13:56:14.285349 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 530c7a7742ff4dc182758c2250c1fabd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 13:56:14.276197 25076 tablet_service.cc:1467] Processing CreateTablet for tablet 17458082ba194fc8b534019c9eab0485 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 13:56:14.309301 25005 tablet_service.cc:1467] Processing CreateTablet for tablet f748bef4f6f64708b4a9fe990a22623f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240501 13:56:14.293357 25151 tablet_service.cc:1467] Processing CreateTablet for tablet e8dff22b19a440d3bf74e5b47c061727 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 13:56:14.299849 25210 tablet_bootstrap.cc:700] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: Time spent opening tablet: real 0.172s user 0.005s sys 0.002s
I20240501 13:56:14.314381 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a49139fcaa8462593981602f13840c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.327558 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e26289285d4f14b97aa17f829a7c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.336063 25007 tablet_service.cc:1467] Processing CreateTablet for tablet 6b63b485dd1847be84f0345d2ec1fb94 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240501 13:56:14.337581 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1fd409e4d541dca63ed7a0544b4c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.350893 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:14.347728 25210 tablet_bootstrap.cc:654] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:14.356220 25058 leader_election.cc:341] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.362290 25209 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:14.362689 24999 tablet_service.cc:1467] Processing CreateTablet for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 13:56:14.364989 25002 tablet_service.cc:1467] Processing CreateTablet for tablet 587937b292d4407094174d4d1e073716 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 13:56:14.362917 25156 tablet_service.cc:1467] Processing CreateTablet for tablet 021330a06d7d4faf9350427cbf308ed9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 13:56:14.364979 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530c7a7742ff4dc182758c2250c1fabd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.364167 25209 ts_tablet_manager.cc:1367] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.279s user 0.026s sys 0.010s
I20240501 13:56:14.369928 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17458082ba194fc8b534019c9eab0485. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.372498 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021330a06d7d4faf9350427cbf308ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.373180 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b63b485dd1847be84f0345d2ec1fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.373842 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587937b292d4407094174d4d1e073716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.374982 25003 tablet_service.cc:1467] Processing CreateTablet for tablet 52f1b26447b54d048d1ce0620e00ef85 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 13:56:14.367822 25083 tablet_service.cc:1467] Processing CreateTablet for tablet b6ae8ea0d9e240c29296a5b826f67196 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 13:56:14.377918 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae8ea0d9e240c29296a5b826f67196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.377987 25209 raft_consensus.cc:348] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.378894 25209 raft_consensus.cc:374] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.379220 25209 raft_consensus.cc:729] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.376271 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f1b26447b54d048d1ce0620e00ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.383721 25207 tablet_bootstrap.cc:654] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.384658 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.387712 25080 tablet_service.cc:1467] Processing CreateTablet for tablet e8dff22b19a440d3bf74e5b47c061727 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 13:56:14.389109 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dff22b19a440d3bf74e5b47c061727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.390424 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 821bad94d33240cd9802ff777bcb842a (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 13:56:14.391758 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821bad94d33240cd9802ff777bcb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.407090 25150 tablet_service.cc:1467] Processing CreateTablet for tablet e059d8fd99574a5eac0b49e9405d7bd4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 13:56:14.417313 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:14.419013 25131 leader_election.cc:341] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.394614 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 4e912ea2c2074a20b15c82376861b3f7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 13:56:14.421908 25209 consensus_queue.cc:260] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.424997 25209 ts_tablet_manager.cc:1398] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.056s user 0.003s sys 0.005s
I20240501 13:56:14.432657 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748bef4f6f64708b4a9fe990a22623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.400753 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ae8ea0d9e240c29296a5b826f67196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.438407 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 821bad94d33240cd9802ff777bcb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.439496 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 4e912ea2c2074a20b15c82376861b3f7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 13:56:14.440207 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dff22b19a440d3bf74e5b47c061727. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.444691 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e059d8fd99574a5eac0b49e9405d7bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.448751 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021330a06d7d4faf9350427cbf308ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.448841 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 021330a06d7d4faf9350427cbf308ed9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 13:56:14.450197 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021330a06d7d4faf9350427cbf308ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.450451 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc140167ab54483b3db40ba7a61937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.452651 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e912ea2c2074a20b15c82376861b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.452911 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e912ea2c2074a20b15c82376861b3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.462803 25214 raft_consensus.cc:482] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:14.463358 25214 raft_consensus.cc:504] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.465377 25214 leader_election.cc:288] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:14.468465 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:14.470116 25131 leader_election.cc:341] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.437276 25077 tablet_service.cc:1467] Processing CreateTablet for tablet bdc140167ab54483b3db40ba7a61937f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 13:56:14.471704 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc140167ab54483b3db40ba7a61937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.478353 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:14.480137 25059 leader_election.cc:341] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.480655 25059 leader_election.cc:302] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:14.480993 25004 tablet_service.cc:1467] Processing CreateTablet for tablet e0be3b84fe26485ea767bfa7d40aa018 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 13:56:14.481926 25217 raft_consensus.cc:2726] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:14.482362 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0be3b84fe26485ea767bfa7d40aa018. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.492805 25078 tablet_service.cc:1467] Processing CreateTablet for tablet e059d8fd99574a5eac0b49e9405d7bd4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 13:56:14.494181 25078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e059d8fd99574a5eac0b49e9405d7bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.494473 25083 tablet_service.cc:1467] Processing CreateTablet for tablet 94e26289285d4f14b97aa17f829a7c96 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240501 13:56:14.495708 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e26289285d4f14b97aa17f829a7c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.507052 25000 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab74f4d1c64cafbbc48dc4accf5917 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 13:56:14.508514 25000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab74f4d1c64cafbbc48dc4accf5917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.513988 25148 tablet_service.cc:1467] Processing CreateTablet for tablet 94e26289285d4f14b97aa17f829a7c96 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240501 13:56:14.504554 25209 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:14.532378 25002 tablet_service.cc:1467] Processing CreateTablet for tablet f348136225c444e1a2ed5658a3306eb7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 13:56:14.533764 25002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f348136225c444e1a2ed5658a3306eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.533787 25210 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:14.535642 25210 ts_tablet_manager.cc:1367] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.502s user 0.011s sys 0.037s
I20240501 13:56:14.538466 25210 raft_consensus.cc:348] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.539243 25210 raft_consensus.cc:374] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.539574 25210 raft_consensus.cc:729] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.538940 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e26289285d4f14b97aa17f829a7c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.546797 25209 tablet_bootstrap.cc:654] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.549754 25210 consensus_queue.cc:260] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.552098 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 587937b292d4407094174d4d1e073716 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 13:56:14.553524 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587937b292d4407094174d4d1e073716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.552590 25075 tablet_service.cc:1467] Processing CreateTablet for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 13:56:14.555548 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:14.580123 24983 leader_election.cc:341] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.580807 24983 leader_election.cc:302] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:14.582226 25211 raft_consensus.cc:2726] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:14.582412 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab74f4d1c64cafbbc48dc4accf5917 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 13:56:14.559505 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.560722 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 6b63b485dd1847be84f0345d2ec1fb94 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240501 13:56:14.584308 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b63b485dd1847be84f0345d2ec1fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.561920 25207 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:14.586735 25207 ts_tablet_manager.cc:1367] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.378s user 0.016s sys 0.003s
I20240501 13:56:14.587349 25006 tablet_service.cc:1467] Processing CreateTablet for tablet bf58f127779f4ebd94273cd7fda138cd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240501 13:56:14.587623 25003 tablet_service.cc:1467] Processing CreateTablet for tablet 098d5f02b04442d2acdc15fcb9a7b429 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 13:56:14.589618 25207 raft_consensus.cc:348] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.590477 25207 raft_consensus.cc:374] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.590786 25207 raft_consensus.cc:729] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.591480 25207 consensus_queue.cc:260] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.558552 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 6b63b485dd1847be84f0345d2ec1fb94 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240501 13:56:14.562880 25077 tablet_service.cc:1467] Processing CreateTablet for tablet e0be3b84fe26485ea767bfa7d40aa018 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 13:56:14.593755 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b63b485dd1847be84f0345d2ec1fb94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.593891 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0be3b84fe26485ea767bfa7d40aa018. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.564066 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:14.554926 25153 tablet_service.cc:1467] Processing CreateTablet for tablet f748bef4f6f64708b4a9fe990a22623f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
W20240501 13:56:14.597834 25133 leader_election.cc:341] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.598296 25133 leader_election.cc:302] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:14.599058 25214 raft_consensus.cc:2726] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:14.599722 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748bef4f6f64708b4a9fe990a22623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.601058 25207 ts_tablet_manager.cc:1398] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240501 13:56:14.600728 25001 tablet_service.cc:1467] Processing CreateTablet for tablet 4c016fa6d9a34be48e77cc49c8f93c44 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 13:56:14.564273 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 52f1b26447b54d048d1ce0620e00ef85 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 13:56:14.605696 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f1b26447b54d048d1ce0620e00ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.606328 24999 tablet_service.cc:1467] Processing CreateTablet for tablet b6e1a6d9d3d9424eb9fb5a781214940e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 13:56:14.571818 25152 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40490 (request call id 25) took 1046 ms. Trace:
I20240501 13:56:14.608000 25207 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:14.607967 25152 rpcz_store.cc:276] 0501 13:56:13.525689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.143178 (+617489us) service_pool.cc:225] Handling call
0501 13:56:14.184120 (+ 40942us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.184221 (+ 101us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.502561 (+318340us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.571436 (+ 68875us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.571757 (+ 321us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 13:56:14.611176 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 587937b292d4407094174d4d1e073716 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 13:56:14.612495 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587937b292d4407094174d4d1e073716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.561792 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:14.576493 25008 tablet_service.cc:1467] Processing CreateTablet for tablet 7b159610f514496a8a1faf7d78c6c839 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240501 13:56:14.552094 25210 ts_tablet_manager.cc:1398] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.016s user 0.001s sys 0.004s
I20240501 13:56:14.570636 25004 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8b602c225b410792a822c1be9b69e9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 13:56:14.617802 25210 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:14.619693 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e1a6d9d3d9424eb9fb5a781214940e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.620663 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf58f127779f4ebd94273cd7fda138cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.623452 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 52f1b26447b54d048d1ce0620e00ef85 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 13:56:14.577961 25081 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33652 (request call id 27) took 1026 ms. Trace:
I20240501 13:56:14.569396 25076 tablet_service.cc:1467] Processing CreateTablet for tablet f748bef4f6f64708b4a9fe990a22623f (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240501 13:56:14.629873 25079 tablet_service.cc:1467] Processing CreateTablet for tablet f348136225c444e1a2ed5658a3306eb7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 13:56:14.643918 25076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748bef4f6f64708b4a9fe990a22623f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.644212 25211 raft_consensus.cc:482] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:14.644739 25211 raft_consensus.cc:504] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.645639 25150 tablet_service.cc:1467] Processing CreateTablet for tablet e0be3b84fe26485ea767bfa7d40aa018 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 13:56:14.647564 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:14.648250 25093 raft_consensus.cc:2454] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:14.625808 25081 rpcz_store.cc:276] 0501 13:56:13.551452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.115402 (+563950us) service_pool.cc:225] Handling call
0501 13:56:14.220185 (+104783us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.220290 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.387334 (+167044us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.564297 (+176963us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.577873 (+ 13576us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 13:56:14.650830 25133 leader_election.cc:341] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.651355 25133 leader_election.cc:302] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:14.652477 25214 raft_consensus.cc:2726] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:14.653765 24982 leader_election.cc:302] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:14.652241 25081 tablet_service.cc:1467] Processing CreateTablet for tablet bf58f127779f4ebd94273cd7fda138cd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240501 13:56:14.654798 25211 leader_election.cc:288] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:14.655331 25211 raft_consensus.cc:2781] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:14.655684 25211 raft_consensus.cc:482] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:14.656008 25211 raft_consensus.cc:3037] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:14.657774 25210 tablet_bootstrap.cc:654] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.657927 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf58f127779f4ebd94273cd7fda138cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.664728 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab74f4d1c64cafbbc48dc4accf5917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.664847 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b159610f514496a8a1faf7d78c6c839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.665457 25003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098d5f02b04442d2acdc15fcb9a7b429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.667645 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8b602c225b410792a822c1be9b69e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.604966 25151 tablet_service.cc:1467] Processing CreateTablet for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 13:56:14.668519 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c016fa6d9a34be48e77cc49c8f93c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.677093 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f348136225c444e1a2ed5658a3306eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.680500 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:14.681979 25150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0be3b84fe26485ea767bfa7d40aa018. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:56:14.681989 24983 leader_election.cc:341] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.684965 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f1b26447b54d048d1ce0620e00ef85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.685909 25211 raft_consensus.cc:504] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.686359 25005 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3ad40b4e34f969bec20bee2bc0162 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 13:56:14.687681 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3ad40b4e34f969bec20bee2bc0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.689186 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:14.690477 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 7b159610f514496a8a1faf7d78c6c839 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20240501 13:56:14.691198 24983 leader_election.cc:341] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.691962 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b159610f514496a8a1faf7d78c6c839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.704716 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1c01c0dc4a345bbb6ce50e6f4ea3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.693976 25155 tablet_service.cc:1467] Processing CreateTablet for tablet 5bab74f4d1c64cafbbc48dc4accf5917 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 13:56:14.696030 25156 tablet_service.cc:1467] Processing CreateTablet for tablet f348136225c444e1a2ed5658a3306eb7 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 13:56:14.709818 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bab74f4d1c64cafbbc48dc4accf5917. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.710654 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 872135f76cb04f6ebfce1899ed32a5a0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 13:56:14.711937 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 872135f76cb04f6ebfce1899ed32a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.711715 25007 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8ee2e10bbd483e9356972bab5ea084 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 13:56:14.713047 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f348136225c444e1a2ed5658a3306eb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.713049 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8ee2e10bbd483e9356972bab5ea084. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:56:14.721643 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:14.722726 25207 tablet_bootstrap.cc:654] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.749469 24999 tablet_service.cc:1467] Processing CreateTablet for tablet b085a215235d482e86b0a9529458b3c6 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 13:56:14.750998 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b085a215235d482e86b0a9529458b3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.739581 25211 leader_election.cc:288] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:14.780808 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8b602c225b410792a822c1be9b69e9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 13:56:14.782143 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8b602c225b410792a822c1be9b69e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.782871 25074 tablet_service.cc:1467] Processing CreateTablet for tablet 098d5f02b04442d2acdc15fcb9a7b429 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 13:56:14.784142 25074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098d5f02b04442d2acdc15fcb9a7b429. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:56:14.770265 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 116585us
I20240501 13:56:14.770649 25211 raft_consensus.cc:482] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:14.801443 25211 raft_consensus.cc:504] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.805446 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:14.807390 24982 leader_election.cc:341] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.809182 25083 tablet_service.cc:1467] Processing CreateTablet for tablet 7b159610f514496a8a1faf7d78c6c839 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240501 13:56:14.810711 25083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b159610f514496a8a1faf7d78c6c839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.813968 25149 tablet_service.cc:1467] Processing CreateTablet for tablet 098d5f02b04442d2acdc15fcb9a7b429 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 13:56:14.815771 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098d5f02b04442d2acdc15fcb9a7b429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.818531 25148 tablet_service.cc:1467] Processing CreateTablet for tablet bf58f127779f4ebd94273cd7fda138cd (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240501 13:56:14.801151 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:14.823519 25093 raft_consensus.cc:3037] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:14.824297 25210 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:14.824849 25210 ts_tablet_manager.cc:1367] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.207s user 0.021s sys 0.009s
I20240501 13:56:14.828398 25209 tablet_bootstrap.cc:492] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:14.829311 25209 ts_tablet_manager.cc:1367] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.369s user 0.034s sys 0.042s
I20240501 13:56:14.834594 25207 tablet_bootstrap.cc:492] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:14.835321 25207 ts_tablet_manager.cc:1367] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.227s user 0.015s sys 0.006s
I20240501 13:56:14.838328 25207 raft_consensus.cc:348] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.840505 25207 raft_consensus.cc:374] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.841070 25207 raft_consensus.cc:729] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.842336 25207 consensus_queue.cc:260] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.839783 25209 raft_consensus.cc:348] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.836841 25157 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8b602c225b410792a822c1be9b69e9 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 13:56:14.848095 25207 ts_tablet_manager.cc:1398] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 13:56:14.843621 25209 raft_consensus.cc:374] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.849555 25209 raft_consensus.cc:729] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
W20240501 13:56:14.851045 24902 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.23.70.131:40317, user_credentials={real_user=slave}} blocked reactor thread for 54054.7us
I20240501 13:56:14.850858 25153 tablet_service.cc:1467] Processing CreateTablet for tablet 4c016fa6d9a34be48e77cc49c8f93c44 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 13:56:14.851284 25209 consensus_queue.cc:260] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.854461 25156 tablet_service.cc:1467] Processing CreateTablet for tablet b6e1a6d9d3d9424eb9fb5a781214940e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 13:56:14.856248 25148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf58f127779f4ebd94273cd7fda138cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.856943 25211 leader_election.cc:288] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:14.857617 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:14.858548 25209 ts_tablet_manager.cc:1398] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240501 13:56:14.859830 25209 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:14.858942 25207 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
W20240501 13:56:14.862345 24983 leader_election.cc:341] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:14.862808 24983 leader_election.cc:302] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:14.865461 25156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e1a6d9d3d9424eb9fb5a781214940e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.866787 25211 raft_consensus.cc:2726] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:14.868211 25211 raft_consensus.cc:482] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:14.868652 25211 raft_consensus.cc:504] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.871886 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:14.872781 25081 tablet_service.cc:1467] Processing CreateTablet for tablet 4c016fa6d9a34be48e77cc49c8f93c44 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 13:56:14.873420 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3ad40b4e34f969bec20bee2bc0162 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 13:56:14.874136 25081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c016fa6d9a34be48e77cc49c8f93c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.833499 25077 tablet_service.cc:1467] Processing CreateTablet for tablet b6e1a6d9d3d9424eb9fb5a781214940e (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 13:56:14.875208 25077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e1a6d9d3d9424eb9fb5a781214940e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.877796 25207 tablet_bootstrap.cc:654] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.878108 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:14.880748 25153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c016fa6d9a34be48e77cc49c8f93c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.884641 25151 tablet_service.cc:1467] Processing CreateTablet for tablet 34d3ad40b4e34f969bec20bee2bc0162 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 13:56:14.885107 25211 leader_election.cc:288] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:14.880102 25076 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33652 (request call id 41) took 1032 ms. Trace:
I20240501 13:56:14.880442 25209 tablet_bootstrap.cc:654] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:14.909781 25167 raft_consensus.cc:2454] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:14.912256 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8b602c225b410792a822c1be9b69e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.915164 25093 raft_consensus.cc:2454] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:14.910596 25082 tablet_service.cc:1467] Processing CreateTablet for tablet 872135f76cb04f6ebfce1899ed32a5a0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 13:56:14.911206 25078 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33652 (request call id 39) took 1121 ms. Trace:
I20240501 13:56:14.827872 25210 raft_consensus.cc:348] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.916610 24983 leader_election.cc:302] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:14.920699 25151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3ad40b4e34f969bec20bee2bc0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.952543 25211 raft_consensus.cc:2781] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:14.953068 25211 raft_consensus.cc:482] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:14.922349 24982 leader_election.cc:302] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:14.953550 25211 raft_consensus.cc:3037] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:14.954531 25233 raft_consensus.cc:2781] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:14.923738 25152 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40490 (request call id 44) took 1013 ms. Trace:
I20240501 13:56:14.958043 25152 rpcz_store.cc:276] 0501 13:56:13.921461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.610264 (+688803us) service_pool.cc:225] Handling call
0501 13:56:14.611993 (+ 1729us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.612098 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.695727 (+ 83629us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.923214 (+227487us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.923679 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 13:56:14.915948 25155 tablet_service.cc:1467] Processing CreateTablet for tablet 872135f76cb04f6ebfce1899ed32a5a0 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 13:56:14.964316 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 872135f76cb04f6ebfce1899ed32a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.966017 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8ee2e10bbd483e9356972bab5ea084 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 13:56:14.925143 25078 rpcz_store.cc:276] 0501 13:56:13.790164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.452450 (+662286us) service_pool.cc:225] Handling call
0501 13:56:14.493699 (+ 41249us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.493797 (+ 98us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.678166 (+184369us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.884921 (+206755us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.911144 (+ 26223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 13:56:14.969800 25211 raft_consensus.cc:504] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.921483 25076 rpcz_store.cc:276] 0501 13:56:13.847533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.568426 (+720893us) service_pool.cc:225] Handling call
0501 13:56:14.643380 (+ 74954us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.643507 (+ 127us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.712203 (+ 68696us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.879592 (+167389us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.880026 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 13:56:14.972710 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:14.929070 25207 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:14.974277 25207 ts_tablet_manager.cc:1367] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.116s user 0.019s sys 0.001s
I20240501 13:56:14.974383 25233 raft_consensus.cc:686] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:14.975327 25233 consensus_queue.cc:237] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.932551 25210 raft_consensus.cc:374] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.976774 25210 raft_consensus.cc:729] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.977234 25207 raft_consensus.cc:348] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:14.977979 25207 raft_consensus.cc:374] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.977604 25210 consensus_queue.cc:260] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.978278 25207 raft_consensus.cc:729] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.978981 25207 consensus_queue.cc:260] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:14.979887 25210 ts_tablet_manager.cc:1398] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.155s user 0.002s sys 0.008s
I20240501 13:56:14.924050 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d3ad40b4e34f969bec20bee2bc0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.980954 25092 raft_consensus.cc:2454] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:14.946374 25150 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40490 (request call id 46) took 1005 ms. Trace:
I20240501 13:56:14.929814 25209 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:14.985894 25209 ts_tablet_manager.cc:1367] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.126s user 0.015s sys 0.000s
I20240501 13:56:14.987608 25210 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:14.988389 25207 ts_tablet_manager.cc:1398] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240501 13:56:14.988767 25209 raft_consensus.cc:348] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.989490 25207 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:14.989627 25209 raft_consensus.cc:374] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:14.989971 25209 raft_consensus.cc:729] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:14.990937 25209 consensus_queue.cc:260] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.948395 25157 tablet_service.cc:1467] Processing CreateTablet for tablet b085a215235d482e86b0a9529458b3c6 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 13:56:14.992703 25082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 872135f76cb04f6ebfce1899ed32a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.994024 25209 ts_tablet_manager.cc:1398] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 13:56:14.994436 25150 rpcz_store.cc:276] 0501 13:56:13.941053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 13:56:14.644480 (+703427us) service_pool.cc:225] Handling call
0501 13:56:14.680152 (+ 35672us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 13:56:14.681620 (+ 1468us) ts_tablet_manager.cc:655] Creating new metadata...
0501 13:56:14.851045 (+169425us) tablet_metadata.cc:641] Metadata flushed
0501 13:56:14.945855 (+ 94810us) tablet_replica.cc:205] Creating consensus instance
0501 13:56:14.946309 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 13:56:14.996537 25157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b085a215235d482e86b0a9529458b3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:14.998272 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a090c154b9467b8e2cad3029c44cee" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:14.998073 24916 catalog_manager.cc:5484] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:14.998888 25167 raft_consensus.cc:3037] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:14.999526 25211 leader_election.cc:288] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.006331 25167 raft_consensus.cc:2454] T 67a090c154b9467b8e2cad3029c44cee P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:15.017501 24983 leader_election.cc:302] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:15.018653 25233 raft_consensus.cc:2781] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.019129 25233 raft_consensus.cc:686] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:15.019609 25210 tablet_bootstrap.cc:654] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.019904 25233 consensus_queue.cc:237] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:14.945466 25079 tablet_service.cc:1467] Processing CreateTablet for tablet b085a215235d482e86b0a9529458b3c6 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 13:56:14.922906 25154 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8ee2e10bbd483e9356972bab5ea084 (DEFAULT_TABLE table=test-workload [id=bfde8915414949bb8bb4162b3ad7533b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 13:56:15.008481 25207 tablet_bootstrap.cc:654] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.025527 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8ee2e10bbd483e9356972bab5ea084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:15.016500 25209 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.016156 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8ee2e10bbd483e9356972bab5ea084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:15.037442 25209 tablet_bootstrap.cc:654] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.042912 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b085a215235d482e86b0a9529458b3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:15.044260 25207 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.045172 25207 ts_tablet_manager.cc:1367] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.007s
I20240501 13:56:15.048977 25207 raft_consensus.cc:348] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.050124 25207 raft_consensus.cc:374] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.050501 25207 raft_consensus.cc:729] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.052893 25207 consensus_queue.cc:260] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:15.061633 25119 tablet.cc:2369] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:56:15.065402 25207 ts_tablet_manager.cc:1398] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.020s user 0.008s sys 0.002s
I20240501 13:56:15.066481 25207 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.067193 25210 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.067680 25210 ts_tablet_manager.cc:1367] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.009s
I20240501 13:56:15.070593 25210 raft_consensus.cc:348] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.071401 25210 raft_consensus.cc:374] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.071698 25210 raft_consensus.cc:729] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.073262 25207 tablet_bootstrap.cc:654] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.089892 25209 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.090628 25209 ts_tablet_manager.cc:1367] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.074s user 0.019s sys 0.011s
I20240501 13:56:15.087955 25210 consensus_queue.cc:260] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.092700 25207 tablet_bootstrap.cc:492] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.093320 25207 ts_tablet_manager.cc:1367] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240501 13:56:15.093544 25209 raft_consensus.cc:348] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.094619 25209 raft_consensus.cc:374] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.095001 25209 raft_consensus.cc:729] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.093583 25210 ts_tablet_manager.cc:1398] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.026s user 0.004s sys 0.004s
I20240501 13:56:15.096308 25207 raft_consensus.cc:348] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.097230 25207 raft_consensus.cc:374] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.097623 25207 raft_consensus.cc:729] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.095921 25209 consensus_queue.cc:260] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.098317 25207 consensus_queue.cc:260] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.099229 25210 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.100781 25207 ts_tablet_manager.cc:1398] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20240501 13:56:15.101850 25207 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.108713 25217 raft_consensus.cc:482] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.108958 25209 ts_tablet_manager.cc:1398] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.018s user 0.004s sys 0.005s
I20240501 13:56:15.109282 25217 raft_consensus.cc:504] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.110033 25209 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.110425 25210 tablet_bootstrap.cc:654] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.111536 25217 leader_election.cc:288] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.114565 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8be04ae2de44a7afa4abbcc4ec3bf5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.115171 25207 tablet_bootstrap.cc:654] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.115319 25018 raft_consensus.cc:2454] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.117532 25209 tablet_bootstrap.cc:654] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.117327 25059 leader_election.cc:302] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:15.118268 25217 raft_consensus.cc:2781] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.118688 25217 raft_consensus.cc:482] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.119127 25217 raft_consensus.cc:3037] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.131582 25217 raft_consensus.cc:504] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.137979 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8be04ae2de44a7afa4abbcc4ec3bf5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.138958 25167 raft_consensus.cc:2454] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.141779 25217 leader_election.cc:288] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.143268 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8be04ae2de44a7afa4abbcc4ec3bf5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:15.143965 25166 raft_consensus.cc:3037] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.144922 25210 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.145372 25210 ts_tablet_manager.cc:1367] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.024s
I20240501 13:56:15.144840 25209 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.145933 25209 ts_tablet_manager.cc:1367] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.006s
I20240501 13:56:15.147413 25210 raft_consensus.cc:348] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.147971 25210 raft_consensus.cc:374] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.141635 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8be04ae2de44a7afa4abbcc4ec3bf5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:15.148231 25210 raft_consensus.cc:729] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.148604 25018 raft_consensus.cc:3037] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.148716 25209 raft_consensus.cc:348] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.149008 25210 consensus_queue.cc:260] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.149506 25209 raft_consensus.cc:374] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.149842 25209 raft_consensus.cc:729] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.150676 25209 consensus_queue.cc:260] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.151243 25210 ts_tablet_manager.cc:1398] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 13:56:15.151556 25233 raft_consensus.cc:482] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.152267 25210 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.151983 25233 raft_consensus.cc:504] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.154389 25233 leader_election.cc:288] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.154664 25209 ts_tablet_manager.cc:1398] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
I20240501 13:56:15.156612 25209 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.156749 25214 raft_consensus.cc:482] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.157227 25214 raft_consensus.cc:504] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.158704 25210 tablet_bootstrap.cc:654] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.159307 25214 leader_election.cc:288] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.161332 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.162946 24916 catalog_manager.cc:5484] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:15.164682 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3add9205fa474b098334f36bc4cb3eef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.177098 25092 raft_consensus.cc:2379] T 3add9205fa474b098334f36bc4cb3eef P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:15.166524 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.167492 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.169937 25209 tablet_bootstrap.cc:654] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.178889 25131 leader_election.cc:302] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.169937 25166 raft_consensus.cc:2454] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
W20240501 13:56:15.182144 24982 leader_election.cc:341] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.183544 25058 leader_election.cc:302] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:15.185573 25214 raft_consensus.cc:3037] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.187139 25217 raft_consensus.cc:2781] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.178913 25167 raft_consensus.cc:2454] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:15.195356 24983 leader_election.cc:302] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:15.196595 25233 raft_consensus.cc:2781] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.197870 25233 raft_consensus.cc:482] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.198364 25233 raft_consensus.cc:3037] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.202589 25211 raft_consensus.cc:482] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.203119 25211 raft_consensus.cc:504] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.203673 25018 raft_consensus.cc:2454] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:15.205703 25211 leader_election.cc:288] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.209976 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:15.212047 24983 leader_election.cc:341] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.207919 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.212379 25217 raft_consensus.cc:686] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:15.212172 25233 raft_consensus.cc:504] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.213542 25093 raft_consensus.cc:2454] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:15.215329 25233 leader_election.cc:288] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.215783 25217 consensus_queue.cc:237] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.217355 25214 raft_consensus.cc:2726] T 3add9205fa474b098334f36bc4cb3eef P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.217523 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:15.218147 25207 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.218688 25207 ts_tablet_manager.cc:1367] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.117s user 0.021s sys 0.038s
I20240501 13:56:15.221280 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:15.221909 25167 raft_consensus.cc:3037] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.222031 25207 raft_consensus.cc:348] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.222803 25207 raft_consensus.cc:374] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.223120 25210 tablet_bootstrap.cc:492] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.222611 24982 leader_election.cc:302] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.223546 25207 raft_consensus.cc:729] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.224074 25233 raft_consensus.cc:2781] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.224633 25233 raft_consensus.cc:482] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.224392 25207 consensus_queue.cc:260] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.225081 25233 raft_consensus.cc:3037] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.223646 25210 ts_tablet_manager.cc:1367] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.072s user 0.029s sys 0.024s
I20240501 13:56:15.226534 25207 ts_tablet_manager.cc:1398] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 13:56:15.229017 25210 raft_consensus.cc:348] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:15.229506 24982 leader_election.cc:341] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.229867 25210 raft_consensus.cc:374] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.230224 25210 raft_consensus.cc:729] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.230280 25167 raft_consensus.cc:2454] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:15.231580 24983 leader_election.cc:302] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:15.233038 25211 raft_consensus.cc:2781] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.233758 25211 raft_consensus.cc:686] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:15.234606 25211 consensus_queue.cc:237] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.235728 25207 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.239936 25209 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.240701 25209 ts_tablet_manager.cc:1367] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.084s user 0.022s sys 0.011s
I20240501 13:56:15.243184 25207 tablet_bootstrap.cc:654] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.243534 24916 catalog_manager.cc:5484] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:15.244577 25233 raft_consensus.cc:504] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.248250 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:15.249603 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
W20240501 13:56:15.250154 24983 leader_election.cc:341] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.250437 25093 raft_consensus.cc:3037] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.253422 25210 consensus_queue.cc:260] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.256141 25210 ts_tablet_manager.cc:1398] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20240501 13:56:15.257236 25210 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.259222 25093 raft_consensus.cc:2454] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:15.260603 24982 leader_election.cc:302] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.261740 25239 raft_consensus.cc:2781] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.262230 25239 raft_consensus.cc:686] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:15.263090 25239 consensus_queue.cc:237] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.263881 25210 tablet_bootstrap.cc:654] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.243748 25209 raft_consensus.cc:348] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.275521 25209 raft_consensus.cc:374] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.277107 25209 raft_consensus.cc:729] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.276146 24913 catalog_manager.cc:5484] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:15.276511 25233 leader_election.cc:288] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.278923 25210 tablet_bootstrap.cc:492] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.279450 25210 ts_tablet_manager.cc:1367] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.000s
I20240501 13:56:15.278421 25209 consensus_queue.cc:260] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.283273 25210 raft_consensus.cc:348] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.284190 25210 raft_consensus.cc:374] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.284622 25210 raft_consensus.cc:729] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.285833 25210 consensus_queue.cc:260] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.283944 25209 ts_tablet_manager.cc:1398] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.043s user 0.005s sys 0.001s
I20240501 13:56:15.288115 25209 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.292192 25210 ts_tablet_manager.cc:1398] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20240501 13:56:15.293238 25210 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.300024 25210 tablet_bootstrap.cc:654] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.313010 24916 catalog_manager.cc:5484] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:15.320549 25210 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.321075 25210 ts_tablet_manager.cc:1367] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s
I20240501 13:56:15.324926 25209 tablet_bootstrap.cc:654] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.323925 25210 raft_consensus.cc:348] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.325352 25210 raft_consensus.cc:374] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.325626 25210 raft_consensus.cc:729] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.326431 25210 consensus_queue.cc:260] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.331918 25207 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.331904 25210 ts_tablet_manager.cc:1398] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20240501 13:56:15.332527 25207 ts_tablet_manager.cc:1367] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.097s user 0.029s sys 0.060s
I20240501 13:56:15.333253 25210 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.335620 25207 raft_consensus.cc:348] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.336501 25207 raft_consensus.cc:374] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.336813 25207 raft_consensus.cc:729] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.337538 25207 consensus_queue.cc:260] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.339830 25210 tablet_bootstrap.cc:654] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.344810 25207 ts_tablet_manager.cc:1398] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240501 13:56:15.346103 25207 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.351547 25210 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.352154 25207 tablet_bootstrap.cc:654] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.352125 25210 ts_tablet_manager.cc:1367] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240501 13:56:15.355898 25210 raft_consensus.cc:348] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.357122 25210 raft_consensus.cc:374] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.357585 25210 raft_consensus.cc:729] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.359709 25210 consensus_queue.cc:260] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.364861 25210 ts_tablet_manager.cc:1398] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.006s sys 0.002s
I20240501 13:56:15.367599 25210 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.371816 25207 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.372414 25207 ts_tablet_manager.cc:1367] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.007s
I20240501 13:56:15.374969 25210 tablet_bootstrap.cc:654] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.375411 25207 raft_consensus.cc:348] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.376166 25207 raft_consensus.cc:374] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.376468 25207 raft_consensus.cc:729] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.377218 25207 consensus_queue.cc:260] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.384971 25207 ts_tablet_manager.cc:1398] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240501 13:56:15.386256 25207 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.398696 25239 raft_consensus.cc:482] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.399411 25239 raft_consensus.cc:504] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.404286 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.404639 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:15.406251 24983 leader_election.cc:341] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.407013 25239 leader_election.cc:288] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:15.409067 24982 leader_election.cc:341] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.409907 25207 tablet_bootstrap.cc:654] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.410339 24982 leader_election.cc:302] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.411181 25210 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.411422 25239 raft_consensus.cc:2726] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.411767 25210 ts_tablet_manager.cc:1367] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.018s
I20240501 13:56:15.415871 25210 raft_consensus.cc:348] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.417361 25210 raft_consensus.cc:374] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.417807 25210 raft_consensus.cc:729] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.419004 25210 consensus_queue.cc:260] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.422401 25210 ts_tablet_manager.cc:1398] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20240501 13:56:15.424736 25210 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.425873 25207 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.426411 25207 ts_tablet_manager.cc:1367] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.008s
I20240501 13:56:15.427389 25239 raft_consensus.cc:482] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.428020 25239 raft_consensus.cc:504] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.430316 25239 leader_election.cc:288] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.432773 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:15.434286 24983 leader_election.cc:341] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.435549 25210 tablet_bootstrap.cc:654] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.437145 25207 raft_consensus.cc:348] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.437883 25207 raft_consensus.cc:374] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.438158 25207 raft_consensus.cc:729] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.439095 25207 consensus_queue.cc:260] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.441622 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:15.443483 24982 leader_election.cc:341] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.444095 24982 leader_election.cc:302] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.445221 25239 raft_consensus.cc:2726] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.451191 25207 ts_tablet_manager.cc:1398] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20240501 13:56:15.452517 25207 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.460795 25214 raft_consensus.cc:482] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.461056 25207 tablet_bootstrap.cc:654] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.461324 25214 raft_consensus.cc:504] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.462663 25209 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.463152 25209 ts_tablet_manager.cc:1367] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.175s user 0.047s sys 0.064s
I20240501 13:56:15.464807 25214 leader_election.cc:288] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.464962 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:15.469018 25133 leader_election.cc:341] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.466090 25209 raft_consensus.cc:348] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.466497 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.469769 25209 raft_consensus.cc:374] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.470152 25209 raft_consensus.cc:729] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.470101 25093 raft_consensus.cc:2454] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:15.471241 25209 consensus_queue.cc:260] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.471949 25131 leader_election.cc:302] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.473376 25209 ts_tablet_manager.cc:1398] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240501 13:56:15.474152 25214 raft_consensus.cc:2781] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.474660 25214 raft_consensus.cc:482] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.475075 25214 raft_consensus.cc:3037] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.488085 25209 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.493187 25210 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.494537 25209 tablet_bootstrap.cc:654] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.495180 25210 ts_tablet_manager.cc:1367] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.071s user 0.022s sys 0.013s
I20240501 13:56:15.499228 25210 raft_consensus.cc:348] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.502774 25210 raft_consensus.cc:374] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.503054 25210 raft_consensus.cc:729] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.503623 25210 consensus_queue.cc:260] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.505172 25207 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.505610 25207 ts_tablet_manager.cc:1367] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.040s
I20240501 13:56:15.505645 25210 ts_tablet_manager.cc:1398] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240501 13:56:15.500628 25214 raft_consensus.cc:504] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.508185 25209 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.508246 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:15.508740 25209 ts_tablet_manager.cc:1367] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.005s
I20240501 13:56:15.508515 25207 raft_consensus.cc:348] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.509169 25207 raft_consensus.cc:374] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.509440 25207 raft_consensus.cc:729] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
W20240501 13:56:15.509835 25133 leader_election.cc:341] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.510136 25207 consensus_queue.cc:260] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.510671 25093 raft_consensus.cc:3037] T 67a090c154b9467b8e2cad3029c44cee P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.511444 25214 leader_election.cc:288] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.511756 25209 raft_consensus.cc:348] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.512523 25210 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.512480 25209 raft_consensus.cc:374] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.512919 25209 raft_consensus.cc:729] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.513548 25209 consensus_queue.cc:260] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.514127 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:15.514703 25092 raft_consensus.cc:3037] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.515792 25239 raft_consensus.cc:482] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.516250 25239 raft_consensus.cc:504] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.519081 25210 tablet_bootstrap.cc:654] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.520792 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:15.522193 24983 leader_election.cc:341] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.520697 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.523370 25092 raft_consensus.cc:2454] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
W20240501 13:56:15.523803 24982 leader_election.cc:341] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.524899 24982 leader_election.cc:302] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.524876 25131 leader_election.cc:302] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.525902 25211 raft_consensus.cc:2726] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.526547 25214 raft_consensus.cc:2781] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.526893 25207 ts_tablet_manager.cc:1398] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240501 13:56:15.527009 25214 raft_consensus.cc:686] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:15.527798 25207 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.527832 25214 consensus_queue.cc:237] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.529886 25239 leader_election.cc:288] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.530002 25209 ts_tablet_manager.cc:1398] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240501 13:56:15.531056 25209 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.534066 25217 raft_consensus.cc:482] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.534543 25217 raft_consensus.cc:504] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.536571 25217 leader_election.cc:288] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.538630 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.539371 25167 raft_consensus.cc:2454] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.540702 25058 leader_election.cc:302] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:15.541059 25239 consensus_queue.cc:1035] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:15.542733 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.543233 25217 raft_consensus.cc:2781] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.545464 25217 raft_consensus.cc:482] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.545820 25217 raft_consensus.cc:3037] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.546530 25207 tablet_bootstrap.cc:654] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.547000 25209 tablet_bootstrap.cc:654] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:15.545007 25059 leader_election.cc:341] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.549367 25210 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.549841 25210 ts_tablet_manager.cc:1367] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.015s
I20240501 13:56:15.552946 25210 raft_consensus.cc:348] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.553645 25210 raft_consensus.cc:374] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.553889 25210 raft_consensus.cc:729] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.554581 25210 consensus_queue.cc:260] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.558357 25209 tablet_bootstrap.cc:492] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.558768 25207 tablet_bootstrap.cc:492] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.558833 25209 ts_tablet_manager.cc:1367] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.003s
I20240501 13:56:15.558954 25255 raft_consensus.cc:482] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.559279 25210 ts_tablet_manager.cc:1398] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:56:15.559341 25207 ts_tablet_manager.cc:1367] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.000s
I20240501 13:56:15.559489 25255 raft_consensus.cc:504] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.560315 25210 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.561834 25209 raft_consensus.cc:348] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.562522 25209 raft_consensus.cc:374] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.562768 25209 raft_consensus.cc:729] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.562737 25207 raft_consensus.cc:348] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.563673 25207 raft_consensus.cc:374] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.563637 25209 consensus_queue.cc:260] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.564031 25207 raft_consensus.cc:729] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.564639 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.564874 25207 consensus_queue.cc:260] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:15.565819 25059 leader_election.cc:341] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.565938 25209 ts_tablet_manager.cc:1398] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 13:56:15.566486 25210 tablet_bootstrap.cc:654] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.563161 25255 leader_election.cc:288] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.566921 25209 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.567240 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.572633 24913 catalog_manager.cc:5484] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:15.573186 25239 consensus_queue.cc:1035] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 13:56:15.575016 25058 leader_election.cc:341] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.575546 25207 ts_tablet_manager.cc:1398] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240501 13:56:15.575942 25058 leader_election.cc:302] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.576692 25207 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.577136 25255 raft_consensus.cc:2726] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.579670 25209 tablet_bootstrap.cc:654] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.581341 25210 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:15.581804 25210 ts_tablet_manager.cc:1367] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.012s
I20240501 13:56:15.583258 25255 raft_consensus.cc:482] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.583665 25207 tablet_bootstrap.cc:654] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.583732 25255 raft_consensus.cc:504] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.586799 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:15.588593 25059 leader_election.cc:341] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.589166 25211 consensus_queue.cc:1035] T 3add9205fa474b098334f36bc4cb3eef P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:15.590160 25255 leader_election.cc:288] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.592437 25209 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.592909 25209 ts_tablet_manager.cc:1367] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20240501 13:56:15.592579 25210 raft_consensus.cc:348] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.593235 25210 raft_consensus.cc:374] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.593147 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.593638 25210 raft_consensus.cc:729] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.593876 25166 raft_consensus.cc:2454] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.594584 25210 consensus_queue.cc:260] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.595631 25058 leader_election.cc:302] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.599347 25207 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.599531 25239 consensus_queue.cc:1035] T 67a090c154b9467b8e2cad3029c44cee P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:15.595911 25209 raft_consensus.cc:348] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.600710 25209 raft_consensus.cc:374] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.601114 25209 raft_consensus.cc:729] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.601951 25255 raft_consensus.cc:2781] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.602336 25255 raft_consensus.cc:482] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.601809 25209 consensus_queue.cc:260] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.602674 25255 raft_consensus.cc:3037] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.604637 25209 ts_tablet_manager.cc:1398] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20240501 13:56:15.605844 25209 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.600003 25207 ts_tablet_manager.cc:1367] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.006s
I20240501 13:56:15.611867 25209 tablet_bootstrap.cc:654] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.611611 25207 raft_consensus.cc:348] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.611395 25217 raft_consensus.cc:504] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.612249 25207 raft_consensus.cc:374] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.620188 25207 raft_consensus.cc:729] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.621942 25207 consensus_queue.cc:260] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.625944 25207 ts_tablet_manager.cc:1398] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240501 13:56:15.626981 25207 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:15.628401 25214 raft_consensus.cc:482] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.628865 25214 raft_consensus.cc:504] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.631006 25214 leader_election.cc:288] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.632789 25207 tablet_bootstrap.cc:654] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.633694 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.634454 25018 raft_consensus.cc:2454] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:15.635373 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.635998 25091 raft_consensus.cc:2454] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:15.637514 25133 leader_election.cc:302] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:15.639295 25214 raft_consensus.cc:2781] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.639706 25214 raft_consensus.cc:482] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.640084 25214 raft_consensus.cc:3037] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.640439 25239 raft_consensus.cc:482] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.641108 25239 raft_consensus.cc:504] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.643755 25239 leader_election.cc:288] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.645320 25211 raft_consensus.cc:482] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.649221 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:15.650815 25217 leader_election.cc:288] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.660928 25281 raft_consensus.cc:482] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.661440 25281 raft_consensus.cc:504] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.661469 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.663272 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d46a9c65a336412d97c25350438ffba9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.664261 25092 raft_consensus.cc:2379] T d46a9c65a336412d97c25350438ffba9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:15.653847 25211 raft_consensus.cc:504] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.676043 24982 leader_election.cc:302] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.677570 25238 raft_consensus.cc:3037] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.676331 25166 raft_consensus.cc:3037] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.694167 25209 tablet_bootstrap.cc:492] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:15.694890 25209 ts_tablet_manager.cc:1367] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.089s user 0.050s sys 0.033s
I20240501 13:56:15.697338 25282 raft_consensus.cc:482] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.697907 25282 raft_consensus.cc:504] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.704919 25217 raft_consensus.cc:482] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.705544 25211 leader_election.cc:288] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.705438 25217 raft_consensus.cc:504] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.697383 25269 raft_consensus.cc:482] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.716980 25209 raft_consensus.cc:348] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.718021 25209 raft_consensus.cc:374] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:15.718379 25209 raft_consensus.cc:729] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.719122 25209 consensus_queue.cc:260] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.721982 25210 ts_tablet_manager.cc:1398] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.140s user 0.030s sys 0.037s
I20240501 13:56:15.723959 25210 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:15.726972 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.727797 25217 leader_election.cc:288] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.721417 25282 leader_election.cc:288] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.729081 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.729904 25167 raft_consensus.cc:2379] T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:15.731184 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.717255 25269 raft_consensus.cc:504] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } }
I20240501 13:56:15.722460 25281 leader_election.cc:288] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.735158 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58e0e1a34b84e78b395e8776f8a48ed" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.744452 25209 ts_tablet_manager.cc:1398] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.049s user 0.012s sys 0.003s
I20240501 13:56:15.745421 25209 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:15.736837 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.757987 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.758637 25016 raft_consensus.cc:2454] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.738823 25282 raft_consensus.cc:482] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.753716 25059 leader_election.cc:302] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.769902 25059 leader_election.cc:302] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:15.770738 25217 raft_consensus.cc:3037] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:15.773609 25058 leader_election.cc:341] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.772193 25282 raft_consensus.cc:504] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:15.767750 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 61576.1us
I20240501 13:56:15.740898 25214 raft_consensus.cc:504] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.784756 25289 raft_consensus.cc:482] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 13:56:15.797797 24983 leader_election.cc:341] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:15.798686 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:15.799302 25015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.773027 25281 consensus_queue.cc:1035] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240501 13:56:15.782219 25255 raft_consensus.cc:504] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.809279 25280 raft_consensus.cc:2781] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.809775 25280 raft_consensus.cc:482] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.810051 25282 leader_election.cc:288] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.810185 25280 raft_consensus.cc:3037] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.811448 25269 leader_election.cc:288] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.812747 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.809337 25167 raft_consensus.cc:2379] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:15.821617 25162 raft_consensus.cc:2454] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.813036 25217 raft_consensus.cc:2726] T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.825394 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:15.814332 25059 leader_election.cc:341] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.814765 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:15.836879 25161 raft_consensus.cc:3037] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.844784 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.845960 25093 raft_consensus.cc:2379] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:15.839007 25291 raft_consensus.cc:482] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.847286 25291 raft_consensus.cc:504] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.855772 25058 leader_election.cc:302] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:15.837941 25059 leader_election.cc:302] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.860215 25281 raft_consensus.cc:2781] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:15.860775 25281 raft_consensus.cc:482] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:15.861196 25281 raft_consensus.cc:3037] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.861429 25282 raft_consensus.cc:2726] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.805644 25289 raft_consensus.cc:504] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.868749 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
W20240501 13:56:15.875161 24982 leader_election.cc:341] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.887580 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c77fc3f2c34f4b58bb2af4583495adba" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.890116 25291 leader_election.cc:288] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:15.890172 25238 raft_consensus.cc:2726] T d46a9c65a336412d97c25350438ffba9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.886991 25167 raft_consensus.cc:3037] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.949079 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:15.934783 25214 leader_election.cc:288] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.892863 24982 leader_election.cc:302] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:15.926895 25291 consensus_queue.cc:1035] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20240501 13:56:15.960166 25209 tablet_bootstrap.cc:654] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:15.950469 25255 leader_election.cc:288] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:15.952356 25059 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.129:33309, user_credentials={real_user=slave}} blocked reactor thread for 140471us
I20240501 13:56:15.960307 25238 raft_consensus.cc:2726] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.961542 25166 raft_consensus.cc:2454] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
W20240501 13:56:15.965515 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 90388.9us
I20240501 13:56:15.966089 25133 leader_election.cc:302] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.977905 25289 leader_election.cc:288] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:15.969311 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:15.980468 25018 raft_consensus.cc:3037] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:15.980402 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.981170 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:15.981379 25289 consensus_queue.cc:1035] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 13:56:15.981130 25059 leader_election.cc:341] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:15.983394 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:15.983880 25058 leader_election.cc:302] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:15.983937 25015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:15.984936 25255 raft_consensus.cc:2781] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:15.985268 25015 raft_consensus.cc:2454] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:15.985726 25214 raft_consensus.cc:2726] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:15.985998 25255 raft_consensus.cc:686] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:15.986956 25255 consensus_queue.cc:237] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.972788 25292 raft_consensus.cc:482] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:15.988097 25292 raft_consensus.cc:504] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:15.990399 25207 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:15.990885 25207 ts_tablet_manager.cc:1367] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.364s user 0.016s sys 0.023s
I20240501 13:56:15.991523 25292 leader_election.cc:288] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:15.996090 25059 leader_election.cc:341] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:16.001323 24983 leader_election.cc:341] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.002627 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:15.993497 25280 raft_consensus.cc:504] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.010440 25207 raft_consensus.cc:348] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.013577 25207 raft_consensus.cc:374] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.013923 25207 raft_consensus.cc:729] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:15.996253 25281 raft_consensus.cc:504] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.017302 25207 consensus_queue.cc:260] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.018503 25210 tablet_bootstrap.cc:654] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.026770 25296 consensus_queue.cc:1035] T e58e0e1a34b84e78b395e8776f8a48ed P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 13:56:16.029390 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.031234 25207 ts_tablet_manager.cc:1398] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.040s user 0.006s sys 0.005s
I20240501 13:56:16.006160 25211 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20240501 13:56:16.034289 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.035693 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.036331 25090 raft_consensus.cc:2379] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:16.039254 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:16.040472 25165 raft_consensus.cc:2379] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:16.036532 25207 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
W20240501 13:56:16.040902 24983 leader_election.cc:341] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.036907 25214 consensus_queue.cc:1035] T d46a9c65a336412d97c25350438ffba9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.032622 25016 raft_consensus.cc:2454] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
W20240501 13:56:16.045588 25131 leader_election.cc:341] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.044983 25280 leader_election.cc:288] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.051272 25282 consensus_queue.cc:1035] T 8d8be04ae2de44a7afa4abbcc4ec3bf5 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240501 13:56:16.048583 25018 raft_consensus.cc:2454] T 8a816d7705ea4eed878efa007b669731 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:16.063728 25133 leader_election.cc:302] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:16.069131 25214 raft_consensus.cc:2781] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.069280 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a816d7705ea4eed878efa007b669731" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.069550 25214 raft_consensus.cc:482] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.070013 25214 raft_consensus.cc:3037] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.071128 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:16.071779 25165 raft_consensus.cc:3037] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:16.062263 24982 leader_election.cc:341] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.075812 24982 leader_election.cc:302] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.077567 25296 raft_consensus.cc:2726] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 13:56:16.077679 24982 leader_election.cc:341] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.078229 25296 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.066013 25280 raft_consensus.cc:482] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.079140 25280 raft_consensus.cc:504] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.079789 25133 leader_election.cc:302] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:16.080641 25289 raft_consensus.cc:2781] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.081141 25289 raft_consensus.cc:686] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:16.081384 25280 leader_election.cc:288] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.078240 24982 leader_election.cc:302] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.082051 25289 consensus_queue.cc:237] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.086026 25165 raft_consensus.cc:2454] T 7b65b2aa3bc84d22bf7ca8258043cc5f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:16.086764 25281 leader_election.cc:288] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.083896 25167 raft_consensus.cc:2454] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:16.084123 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.087478 25161 raft_consensus.cc:2454] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:16.087954 25058 leader_election.cc:302] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.088960 25058 leader_election.cc:302] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.089114 25281 raft_consensus.cc:2781] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.089792 25217 raft_consensus.cc:2781] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.089932 25058 leader_election.cc:302] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:16.090220 25217 raft_consensus.cc:482] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.090579 25217 raft_consensus.cc:3037] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.086648 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b65b2aa3bc84d22bf7ca8258043cc5f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.092650 25281 raft_consensus.cc:686] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:16.093703 25296 raft_consensus.cc:2726] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.093971 25281 consensus_queue.cc:237] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.093053 25280 raft_consensus.cc:2781] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.082270 25214 raft_consensus.cc:504] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.092142 25207 tablet_bootstrap.cc:654] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:16.106945 25302 log.cc:924] Time spent T e58e0e1a34b84e78b395e8776f8a48ed P ef020bad8d24499281f69de00e90af06: Append to log took a long time: real 0.079s user 0.004s sys 0.000s
I20240501 13:56:16.101236 25018 raft_consensus.cc:3037] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.101179 25280 raft_consensus.cc:686] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:16.166972 25280 consensus_queue.cc:237] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.175477 25214 leader_election.cc:288] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.178378 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:16.178797 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e4ec10c6ef41a9815b71da267be33f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.179647 25017 raft_consensus.cc:3037] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:16.180721 25131 leader_election.cc:341] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.081429 25059 leader_election.cc:302] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
W20240501 13:56:16.219938 25059 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.129:33309, user_credentials={real_user=slave}} blocked reactor thread for 138595us
I20240501 13:56:16.217993 25282 raft_consensus.cc:2726] T 8a816d7705ea4eed878efa007b669731 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.235363 25014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.238189 25014 raft_consensus.cc:2454] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:16.262223 24913 catalog_manager.cc:5484] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:16.251210 25017 raft_consensus.cc:2454] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:16.265847 25217 raft_consensus.cc:504] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.272110 25217 leader_election.cc:288] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.272632 25133 leader_election.cc:302] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:16.274693 25289 raft_consensus.cc:2781] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.275373 25289 raft_consensus.cc:686] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:16.276508 25289 consensus_queue.cc:237] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.294798 25018 raft_consensus.cc:2454] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:16.306248 25210 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.306751 25210 ts_tablet_manager.cc:1367] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.583s user 0.021s sys 0.029s
I20240501 13:56:16.404126 25210 raft_consensus.cc:348] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.404899 25210 raft_consensus.cc:374] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.342852 24916 catalog_manager.cc:5484] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:16.405411 25210 raft_consensus.cc:729] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.403930 25209 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
W20240501 13:56:16.404069 25058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.131:40317, user_credentials={real_user=slave}} blocked reactor thread for 132696us
I20240501 13:56:16.406188 25210 consensus_queue.cc:260] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.407667 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:16.408270 25164 raft_consensus.cc:3037] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:16.388911 25308 log.cc:924] Time spent T e58e0e1a34b84e78b395e8776f8a48ed P 171c5cc5f6df4820bc16946f8717737e: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20240501 13:56:16.408874 25210 ts_tablet_manager.cc:1398] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.102s user 0.006s sys 0.005s
I20240501 13:56:16.409709 25210 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
W20240501 13:56:16.410682 25311 log.cc:924] Time spent T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06: Append to log took a long time: real 0.090s user 0.001s sys 0.003s
I20240501 13:56:16.406246 25209 ts_tablet_manager.cc:1367] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.661s user 0.020s sys 0.028s
I20240501 13:56:16.416828 25209 raft_consensus.cc:348] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.417629 25209 raft_consensus.cc:374] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.417915 25209 raft_consensus.cc:729] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.419035 25281 raft_consensus.cc:482] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.421521 25281 raft_consensus.cc:504] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.423496 25281 leader_election.cc:288] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.423795 25207 tablet_bootstrap.cc:492] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:16.424278 25207 ts_tablet_manager.cc:1367] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.388s user 0.015s sys 0.009s
I20240501 13:56:16.424612 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.425042 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:16.426137 25058 leader_election.cc:341] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.352946 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aae555c69934908afd09a93e6191d17" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.426762 25207 raft_consensus.cc:348] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:16.427387 25059 leader_election.cc:341] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.427488 25207 raft_consensus.cc:374] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.427990 25207 raft_consensus.cc:729] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.427892 25059 leader_election.cc:302] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.428822 25018 raft_consensus.cc:3037] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.430974 25255 raft_consensus.cc:2726] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.432794 25210 tablet_bootstrap.cc:654] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.428834 25207 consensus_queue.cc:260] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.435070 25209 consensus_queue.cc:260] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.440594 25164 raft_consensus.cc:2454] T 3aae555c69934908afd09a93e6191d17 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:16.441268 25207 ts_tablet_manager.cc:1398] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20240501 13:56:16.441835 25058 leader_election.cc:302] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.442183 25207 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:16.442981 25209 ts_tablet_manager.cc:1398] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240501 13:56:16.442911 25255 raft_consensus.cc:2781] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.443893 25209 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:16.444581 25255 raft_consensus.cc:686] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:16.445663 25255 consensus_queue.cc:237] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.458503 25207 tablet_bootstrap.cc:654] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.460808 25209 tablet_bootstrap.cc:654] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.465658 25018 raft_consensus.cc:2454] T 3aae555c69934908afd09a93e6191d17 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
W20240501 13:56:16.469550 25059 consensus_peers.cc:485] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e -> Peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Couldn't send request to peer 346b2bb141744d459f1fa3ad602d1a15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:16.474469 25210 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.474958 25210 ts_tablet_manager.cc:1367] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.014s
I20240501 13:56:16.478323 25281 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:16.481086 25211 raft_consensus.cc:482] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.481762 25211 raft_consensus.cc:504] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.482357 25296 raft_consensus.cc:482] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.482298 25210 raft_consensus.cc:348] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.483172 25210 raft_consensus.cc:374] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.482906 25296 raft_consensus.cc:504] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.483428 25210 raft_consensus.cc:729] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.484171 25210 consensus_queue.cc:260] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.484827 25296 leader_election.cc:288] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.486361 25210 ts_tablet_manager.cc:1398] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240501 13:56:16.487255 25210 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:16.487382 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.490208 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.494349 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:16.495910 24983 leader_election.cc:341] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.496269 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5858563bd674a6a8ed86d4b6b174cb2" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.497076 25164 raft_consensus.cc:2379] T f5858563bd674a6a8ed86d4b6b174cb2 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:16.497838 25210 tablet_bootstrap.cc:654] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.494812 25211 leader_election.cc:288] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:16.515156 24982 leader_election.cc:341] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.515864 24982 leader_election.cc:302] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.516652 24983 leader_election.cc:302] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.517550 25211 raft_consensus.cc:2726] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.518163 25296 raft_consensus.cc:3037] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.522810 25207 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:16.521595 24914 catalog_manager.cc:5484] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:16.532608 25296 raft_consensus.cc:2726] T f5858563bd674a6a8ed86d4b6b174cb2 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.523455 25207 ts_tablet_manager.cc:1367] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.081s user 0.012s sys 0.004s
I20240501 13:56:16.536738 25207 raft_consensus.cc:348] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.537459 25207 raft_consensus.cc:374] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.537731 25207 raft_consensus.cc:729] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.538486 25207 consensus_queue.cc:260] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.540751 25207 ts_tablet_manager.cc:1398] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 13:56:16.541473 25207 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:16.546957 25209 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:16.547463 25209 ts_tablet_manager.cc:1367] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.064s
I20240501 13:56:16.547889 24916 catalog_manager.cc:5484] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:16.549214 25207 tablet_bootstrap.cc:654] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.549855 25209 raft_consensus.cc:348] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.550621 25209 raft_consensus.cc:374] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.550947 25209 raft_consensus.cc:729] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.550314 24916 catalog_manager.cc:5484] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:16.552503 25209 consensus_queue.cc:260] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.553090 24916 catalog_manager.cc:5484] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:16.554792 25209 ts_tablet_manager.cc:1398] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 13:56:16.555684 25209 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:16.558626 25210 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.561378 25209 tablet_bootstrap.cc:654] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.561861 25210 ts_tablet_manager.cc:1367] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.075s user 0.017s sys 0.009s
I20240501 13:56:16.567513 25210 raft_consensus.cc:348] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.568574 25210 raft_consensus.cc:374] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.568926 25210 raft_consensus.cc:729] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.569773 25210 consensus_queue.cc:260] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.571389 25280 raft_consensus.cc:482] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.571995 25280 raft_consensus.cc:504] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.575450 25280 leader_election.cc:288] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.576576 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.577597 25210 ts_tablet_manager.cc:1398] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20240501 13:56:16.577535 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:16.577965 25058 leader_election.cc:341] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.578264 25016 raft_consensus.cc:2454] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:16.578927 25210 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:16.579672 25059 leader_election.cc:302] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.580936 25280 raft_consensus.cc:2781] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.581313 25280 raft_consensus.cc:482] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.581640 25280 raft_consensus.cc:3037] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.587337 25210 tablet_bootstrap.cc:654] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.589375 25209 tablet_bootstrap.cc:492] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:16.590062 25209 ts_tablet_manager.cc:1367] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.035s user 0.014s sys 0.020s
I20240501 13:56:16.591207 25280 raft_consensus.cc:504] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.593350 25209 raft_consensus.cc:348] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.594300 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.594463 25209 raft_consensus.cc:374] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.594970 25016 raft_consensus.cc:3037] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.594925 25209 raft_consensus.cc:729] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.598421 25209 consensus_queue.cc:260] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.599635 25269 raft_consensus.cc:482] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.600090 25269 raft_consensus.cc:504] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.600216 25217 consensus_queue.cc:1035] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:16.602190 25269 leader_election.cc:288] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.603717 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.604457 25017 raft_consensus.cc:2454] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:16.607219 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.608009 25090 raft_consensus.cc:2454] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:16.608021 25209 ts_tablet_manager.cc:1398] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240501 13:56:16.609503 25209 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:16.613562 25133 leader_election.cc:302] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.614825 25289 raft_consensus.cc:2781] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.615128 25210 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.615324 25289 raft_consensus.cc:482] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.615629 25210 ts_tablet_manager.cc:1367] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.011s
I20240501 13:56:16.615725 25289 raft_consensus.cc:3037] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.616878 25209 tablet_bootstrap.cc:654] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.619917 25210 raft_consensus.cc:348] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.620680 25210 raft_consensus.cc:374] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.620975 25210 raft_consensus.cc:729] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.621698 25210 consensus_queue.cc:260] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.623780 25210 ts_tablet_manager.cc:1398] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20240501 13:56:16.625445 25210 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:16.626789 25207 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:16.627400 25207 ts_tablet_manager.cc:1367] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.004s
I20240501 13:56:16.632737 25281 consensus_queue.cc:1035] T 7b65b2aa3bc84d22bf7ca8258043cc5f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 13:56:16.641757 25016 raft_consensus.cc:2454] T f5fbdfb5256246c5b6701c5f49c3deb9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:16.643059 25207 raft_consensus.cc:348] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.644775 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5fbdfb5256246c5b6701c5f49c3deb9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:16.644984 25207 raft_consensus.cc:374] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.645547 25207 raft_consensus.cc:729] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
W20240501 13:56:16.646203 25058 leader_election.cc:341] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.646626 25207 consensus_queue.cc:260] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.648955 25207 ts_tablet_manager.cc:1398] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20240501 13:56:16.644146 25280 leader_election.cc:288] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.650614 25207 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:16.656929 25059 leader_election.cc:302] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.666190 25281 raft_consensus.cc:2781] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.666832 25281 raft_consensus.cc:686] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:16.671061 25281 consensus_queue.cc:237] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.673475 25210 tablet_bootstrap.cc:654] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.678381 25209 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:16.678927 25209 ts_tablet_manager.cc:1367] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.022s
I20240501 13:56:16.678493 25289 raft_consensus.cc:504] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.690747 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.691501 25016 raft_consensus.cc:3037] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.692559 25207 tablet_bootstrap.cc:654] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.694504 25209 raft_consensus.cc:348] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.695243 25209 raft_consensus.cc:374] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.695518 25209 raft_consensus.cc:729] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.700278 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba50ad350b4473cb475aa22f2ab76ef" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:16.700939 25090 raft_consensus.cc:3037] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.701684 25091 raft_consensus.cc:3037] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.704464 25016 raft_consensus.cc:2454] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:16.705525 25209 consensus_queue.cc:260] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.706434 25214 consensus_queue.cc:1035] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:16.708550 25133 leader_election.cc:302] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.712731 25294 raft_consensus.cc:2781] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.713285 25294 raft_consensus.cc:686] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:16.714177 25294 consensus_queue.cc:237] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.705317 25289 leader_election.cc:288] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.726838 25214 consensus_queue.cc:1035] T 8a816d7705ea4eed878efa007b669731 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:16.746541 25209 ts_tablet_manager.cc:1398] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.058s user 0.017s sys 0.017s
I20240501 13:56:16.748613 25209 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:16.746012 24913 catalog_manager.cc:5484] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:16.764645 25090 raft_consensus.cc:2454] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:16.765039 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:16.790639 25207 tablet_bootstrap.cc:492] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:16.791044 25207 ts_tablet_manager.cc:1367] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.141s user 0.013s sys 0.048s
I20240501 13:56:16.795912 25207 raft_consensus.cc:348] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.796757 25207 raft_consensus.cc:374] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.797183 25207 raft_consensus.cc:729] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.798105 25207 consensus_queue.cc:260] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.800495 25207 ts_tablet_manager.cc:1398] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240501 13:56:16.801458 25207 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:16.800499 25289 raft_consensus.cc:482] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.806993 25289 raft_consensus.cc:504] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.810207 25289 leader_election.cc:288] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.810794 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c110d748768044dabb2ba88e75ce883b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.815955 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c110d748768044dabb2ba88e75ce883b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.816742 25018 raft_consensus.cc:2454] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:16.812197 24913 catalog_manager.cc:5484] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:16.800648 25294 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.816099 25091 raft_consensus.cc:2454] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:16.814704 25281 consensus_queue.cc:1035] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.819846 25133 leader_election.cc:302] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.832510 25294 raft_consensus.cc:2781] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.839835 25294 raft_consensus.cc:482] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.840229 25294 raft_consensus.cc:3037] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.868999 25210 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.869534 25210 ts_tablet_manager.cc:1367] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.244s user 0.037s sys 0.031s
I20240501 13:56:16.870671 25255 consensus_queue.cc:1035] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.872105 25210 raft_consensus.cc:348] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.872848 25210 raft_consensus.cc:374] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.873118 25210 raft_consensus.cc:729] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.873733 25210 consensus_queue.cc:260] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.875916 25210 ts_tablet_manager.cc:1398] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 13:56:16.877135 25210 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:16.898425 25294 raft_consensus.cc:504] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.899830 25207 tablet_bootstrap.cc:654] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.902132 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c110d748768044dabb2ba88e75ce883b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.902833 25016 raft_consensus.cc:3037] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.903177 25209 tablet_bootstrap.cc:654] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.896445 25296 raft_consensus.cc:482] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.904834 25296 raft_consensus.cc:504] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.907824 25281 consensus_queue.cc:1035] T f5858563bd674a6a8ed86d4b6b174cb2 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.908526 25296 leader_election.cc:288] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.909684 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.911072 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56eceec85d2f42f5a0baf57bb03b45f4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.905702 25214 raft_consensus.cc:482] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.912655 24983 leader_election.cc:302] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:16.912868 25214 raft_consensus.cc:504] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.913825 25211 raft_consensus.cc:2726] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:16.917845 25015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.918414 25214 leader_election.cc:288] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
W20240501 13:56:16.919714 25133 leader_election.cc:341] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.919747 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:16.920897 25093 raft_consensus.cc:2454] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:16.921691 25294 leader_election.cc:288] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.922197 25131 leader_election.cc:302] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:16.930677 25016 raft_consensus.cc:2454] T c110d748768044dabb2ba88e75ce883b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:16.932520 25133 leader_election.cc:302] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.932994 25210 tablet_bootstrap.cc:654] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:16.933665 25214 raft_consensus.cc:2781] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:16.925635 25294 raft_consensus.cc:2781] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.935348 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c110d748768044dabb2ba88e75ce883b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:16.935665 25214 raft_consensus.cc:686] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:16.938041 25093 raft_consensus.cc:3037] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.937348 25294 raft_consensus.cc:482] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.941608 25294 raft_consensus.cc:3037] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.939575 25214 consensus_queue.cc:237] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:16.956473 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:16.974135 25281 raft_consensus.cc:482] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:16.979615 25281 raft_consensus.cc:504] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.981245 25210 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:16.981755 25210 ts_tablet_manager.cc:1367] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.105s user 0.016s sys 0.016s
I20240501 13:56:16.984216 25210 raft_consensus.cc:348] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.985037 25210 raft_consensus.cc:374] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:16.985348 25210 raft_consensus.cc:729] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:16.988085 25281 leader_election.cc:288] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:16.989179 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:16.989892 25166 raft_consensus.cc:2454] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:16.964397 25294 raft_consensus.cc:504] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:16.993974 25058 leader_election.cc:302] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:16.994720 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:16.995304 25281 raft_consensus.cc:2781] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:16.995371 25294 leader_election.cc:288] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:16.995414 25017 raft_consensus.cc:2454] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:16.995824 25281 raft_consensus.cc:482] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:16.996294 25281 raft_consensus.cc:3037] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:16.997017 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:16.997323 25255 consensus_queue.cc:1035] T 3aae555c69934908afd09a93e6191d17 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:16.998239 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:16.998884 25090 raft_consensus.cc:3037] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:17.005234 25133 leader_election.cc:341] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:16.986124 25210 consensus_queue.cc:260] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.009862 25090 raft_consensus.cc:2454] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.010814 25207 tablet_bootstrap.cc:492] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.006785 25093 raft_consensus.cc:2454] T c110d748768044dabb2ba88e75ce883b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.011304 25207 ts_tablet_manager.cc:1367] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.210s user 0.006s sys 0.020s
I20240501 13:56:17.011078 25131 leader_election.cc:302] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:17.010171 25210 ts_tablet_manager.cc:1398] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240501 13:56:17.013339 25294 raft_consensus.cc:2781] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.013602 25210 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.014019 25294 raft_consensus.cc:686] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:17.013810 25207 raft_consensus.cc:348] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.014531 25207 raft_consensus.cc:374] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.014815 25207 raft_consensus.cc:729] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.015537 25207 consensus_queue.cc:260] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.014851 25294 consensus_queue.cc:237] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.017869 25209 tablet_bootstrap.cc:492] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.019002 25209 ts_tablet_manager.cc:1367] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.271s user 0.013s sys 0.031s
I20240501 13:56:17.018514 25281 raft_consensus.cc:504] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.024735 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.025061 25281 leader_election.cc:288] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.025421 25017 raft_consensus.cc:3037] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.026185 25296 raft_consensus.cc:482] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.026654 25296 raft_consensus.cc:504] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.027403 25207 ts_tablet_manager.cc:1398] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240501 13:56:17.028601 25207 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.028910 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b683a5a430741e6ac1949303d905584" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:17.029518 25164 raft_consensus.cc:3037] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.030328 25296 leader_election.cc:288] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.029507 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.031670 25090 raft_consensus.cc:2379] T 70b9d51a2e094f0d8b45171217807a23 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:17.031580 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70b9d51a2e094f0d8b45171217807a23" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.033581 24983 leader_election.cc:302] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.034155 25210 tablet_bootstrap.cc:654] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.034701 25296 raft_consensus.cc:3037] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.023231 24913 catalog_manager.cc:5484] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.036516 25209 raft_consensus.cc:348] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.043880 25209 raft_consensus.cc:374] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.044175 25209 raft_consensus.cc:729] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.044865 25209 consensus_queue.cc:260] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.046976 25209 ts_tablet_manager.cc:1398] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.028s user 0.005s sys 0.005s
I20240501 13:56:17.047823 25209 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.047961 25164 raft_consensus.cc:2454] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.040947 25207 tablet_bootstrap.cc:654] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.049906 25058 leader_election.cc:302] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.051376 25281 raft_consensus.cc:2781] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.051997 25281 raft_consensus.cc:686] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:17.052956 25281 consensus_queue.cc:237] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.042553 25296 raft_consensus.cc:2726] T 70b9d51a2e094f0d8b45171217807a23 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.057590 25209 tablet_bootstrap.cc:654] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.059430 25207 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.059897 25207 ts_tablet_manager.cc:1367] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.000s
I20240501 13:56:17.062331 25207 raft_consensus.cc:348] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.063019 25207 raft_consensus.cc:374] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.063300 25207 raft_consensus.cc:729] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.064194 25207 consensus_queue.cc:260] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.066954 25207 ts_tablet_manager.cc:1398] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 13:56:17.066540 24915 catalog_manager.cc:5484] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.068601 25207 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.040125 25017 raft_consensus.cc:2454] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.082999 25207 tablet_bootstrap.cc:654] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.083685 25016 raft_consensus.cc:3037] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.094919 24916 catalog_manager.cc:5484] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:17.120780 25209 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.121354 25209 ts_tablet_manager.cc:1367] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.074s user 0.036s sys 0.031s
I20240501 13:56:17.121549 25210 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.121991 25210 ts_tablet_manager.cc:1367] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.109s user 0.024s sys 0.025s
I20240501 13:56:17.123992 25209 raft_consensus.cc:348] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.124423 25210 raft_consensus.cc:348] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.124820 25209 raft_consensus.cc:374] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.125128 25210 raft_consensus.cc:374] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.125197 25209 raft_consensus.cc:729] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.126351 25209 consensus_queue.cc:260] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.128741 25209 ts_tablet_manager.cc:1398] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 13:56:17.129768 25209 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.125490 25210 raft_consensus.cc:729] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.135326 25210 consensus_queue.cc:260] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.137079 25210 ts_tablet_manager.cc:1398] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20240501 13:56:17.137981 25210 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.156601 25210 tablet_bootstrap.cc:654] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.156592 25209 tablet_bootstrap.cc:654] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.157670 25207 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.158077 25207 ts_tablet_manager.cc:1367] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.090s user 0.014s sys 0.010s
I20240501 13:56:17.160750 25207 raft_consensus.cc:348] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.161295 25207 raft_consensus.cc:374] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.161540 25207 raft_consensus.cc:729] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.161744 25209 tablet_bootstrap.cc:492] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.162158 25209 ts_tablet_manager.cc:1367] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.001s
I20240501 13:56:17.162696 25210 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.162284 25207 consensus_queue.cc:260] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.163096 25210 ts_tablet_manager.cc:1367] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20240501 13:56:17.164299 25207 ts_tablet_manager.cc:1398] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 13:56:17.165531 25207 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.165627 25210 raft_consensus.cc:348] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.166293 25210 raft_consensus.cc:374] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.166580 25210 raft_consensus.cc:729] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.166422 25209 raft_consensus.cc:348] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.167325 25209 raft_consensus.cc:374] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.167758 25209 raft_consensus.cc:729] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.167274 25210 consensus_queue.cc:260] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.168452 25209 consensus_queue.cc:260] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.169560 25210 ts_tablet_manager.cc:1398] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240501 13:56:17.170485 25210 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.170529 25209 ts_tablet_manager.cc:1398] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20240501 13:56:17.172569 25209 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.176543 25210 tablet_bootstrap.cc:654] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.177543 25209 tablet_bootstrap.cc:654] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.181982 25281 consensus_queue.cc:1035] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.187659 25294 consensus_queue.cc:1035] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.190456 25210 tablet_bootstrap.cc:492] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.190999 25210 ts_tablet_manager.cc:1367] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.006s
I20240501 13:56:17.191468 25164 raft_consensus.cc:3037] T f5fbdfb5256246c5b6701c5f49c3deb9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.194136 25210 raft_consensus.cc:348] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.197180 25210 raft_consensus.cc:374] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.197517 25210 raft_consensus.cc:729] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.200884 25210 consensus_queue.cc:260] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.203038 25210 ts_tablet_manager.cc:1398] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 13:56:17.203908 25210 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.205685 25281 consensus_queue.cc:1035] T f5fbdfb5256246c5b6701c5f49c3deb9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.209744 25210 tablet_bootstrap.cc:654] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.212024 25207 tablet_bootstrap.cc:654] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.196902 25209 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.212654 25209 ts_tablet_manager.cc:1367] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.004s
I20240501 13:56:17.215205 25209 raft_consensus.cc:348] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.215296 25294 consensus_queue.cc:1035] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.215983 25209 raft_consensus.cc:374] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.216284 25209 raft_consensus.cc:729] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.228927 25281 raft_consensus.cc:482] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.229678 25281 raft_consensus.cc:504] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.234431 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.235217 25018 raft_consensus.cc:2454] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:17.236239 25255 raft_consensus.cc:482] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.236763 25255 raft_consensus.cc:504] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.241007 25059 leader_election.cc:302] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:17.243711 25318 raft_consensus.cc:2781] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.244169 25318 raft_consensus.cc:482] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.244678 25318 raft_consensus.cc:3037] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.287917 25214 raft_consensus.cc:482] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.288821 25214 raft_consensus.cc:504] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.297586 25209 consensus_queue.cc:260] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.300172 25214 leader_election.cc:288] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.299836 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.302160 25018 raft_consensus.cc:2454] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:17.299904 25209 ts_tablet_manager.cc:1398] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.087s user 0.029s sys 0.055s
I20240501 13:56:17.304797 25209 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.306684 25255 leader_election.cc:288] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.305223 25280 raft_consensus.cc:482] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.311599 25280 raft_consensus.cc:504] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.313351 25133 leader_election.cc:302] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.320394 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.321322 25294 raft_consensus.cc:2781] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.321380 25280 leader_election.cc:288] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.321820 25294 raft_consensus.cc:482] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.322207 25294 raft_consensus.cc:3037] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:17.322862 25058 leader_election.cc:341] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.324434 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.324635 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.325152 25162 raft_consensus.cc:2454] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
W20240501 13:56:17.326973 25058 leader_election.cc:341] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.329656 25294 raft_consensus.cc:504] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.335966 25296 raft_consensus.cc:482] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.336469 25296 raft_consensus.cc:504] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.336840 25210 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.337472 25210 ts_tablet_manager.cc:1367] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.134s user 0.021s sys 0.007s
I20240501 13:56:17.338349 25296 leader_election.cc:288] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.338438 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.339092 25018 raft_consensus.cc:3037] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.330142 25280 raft_consensus.cc:482] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.340657 25280 raft_consensus.cc:504] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.341037 25210 raft_consensus.cc:348] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.342048 25210 raft_consensus.cc:374] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.342418 25210 raft_consensus.cc:729] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.343108 25210 consensus_queue.cc:260] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.313467 25281 leader_election.cc:288] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.344600 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.350941 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.352128 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.352406 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.352614 25018 raft_consensus.cc:2454] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
W20240501 13:56:17.353255 24983 leader_election.cc:341] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.353855 25162 raft_consensus.cc:2379] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
W20240501 13:56:17.353788 24982 leader_election.cc:341] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.354305 24982 leader_election.cc:302] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.343189 25280 leader_election.cc:288] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.354929 25133 leader_election.cc:302] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.355396 25211 raft_consensus.cc:2726] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.356402 25269 raft_consensus.cc:2781] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.356902 25269 raft_consensus.cc:686] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:17.353262 25093 raft_consensus.cc:2454] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:17.357811 25269 consensus_queue.cc:237] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.365769 25210 ts_tablet_manager.cc:1398] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.028s user 0.008s sys 0.001s
I20240501 13:56:17.366500 25210 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.363950 25294 leader_election.cc:288] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.369939 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.370647 25018 raft_consensus.cc:2454] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:17.372871 25294 consensus_queue.cc:1035] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.373072 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:17.373950 25093 raft_consensus.cc:3037] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.383591 25210 tablet_bootstrap.cc:654] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.383298 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.384074 25018 raft_consensus.cc:2454] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:17.386852 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ec4e2e661b4705afa2e4a2c4a76bd5" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.387243 25209 tablet_bootstrap.cc:654] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.386754 25294 consensus_queue.cc:1035] T c110d748768044dabb2ba88e75ce883b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.387802 25018 raft_consensus.cc:2379] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:17.388856 25059 leader_election.cc:302] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.389979 25280 raft_consensus.cc:2781] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.390373 25280 raft_consensus.cc:482] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.390758 25280 raft_consensus.cc:3037] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.390899 25059 leader_election.cc:302] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.394481 25255 raft_consensus.cc:2781] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.394963 25255 raft_consensus.cc:482] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.395376 25255 raft_consensus.cc:3037] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.401000 25280 raft_consensus.cc:504] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.408252 25210 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.417280 25210 ts_tablet_manager.cc:1367] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.017s
I20240501 13:56:17.418128 25318 raft_consensus.cc:504] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.408731 25207 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.415264 25269 consensus_queue.cc:1035] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.420594 25059 leader_election.cc:302] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.420497 24913 catalog_manager.cc:5484] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.423697 25280 leader_election.cc:288] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.420543 25210 raft_consensus.cc:348] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.424780 25210 raft_consensus.cc:374] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.424906 25093 raft_consensus.cc:2454] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.424973 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:17.425730 25207 ts_tablet_manager.cc:1367] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.260s user 0.009s sys 0.017s
I20240501 13:56:17.425756 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.426450 25016 raft_consensus.cc:3037] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.428761 25280 raft_consensus.cc:482] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.425091 25210 raft_consensus.cc:729] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.429308 25280 raft_consensus.cc:504] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.430279 25210 consensus_queue.cc:260] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.436249 25016 raft_consensus.cc:2454] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
W20240501 13:56:17.430747 25058 leader_election.cc:341] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.432472 25280 leader_election.cc:288] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.438015 25210 ts_tablet_manager.cc:1398] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 13:56:17.438762 25210 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.440165 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:17.441609 25058 leader_election.cc:341] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.445015 25059 leader_election.cc:302] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:17.446691 25280 raft_consensus.cc:2781] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.447419 25280 raft_consensus.cc:686] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:17.438759 25255 raft_consensus.cc:504] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.450064 25318 leader_election.cc:288] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.452133 25255 leader_election.cc:288] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.452322 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.448400 25210 tablet_bootstrap.cc:654] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.453032 25017 raft_consensus.cc:3037] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.456466 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
W20240501 13:56:17.458024 25058 leader_election.cc:341] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.458935 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73740a9c3d7443cea7a555b0367a2b66" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:17.459587 25164 raft_consensus.cc:3037] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.459966 25017 raft_consensus.cc:2454] T 357df45144824ef386e687e270aa8e77 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.449193 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.448329 25280 consensus_queue.cc:237] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.448302 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55e60d3e56d348209c02fb051fd3d27f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.465229 25018 raft_consensus.cc:3037] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.470943 25059 leader_election.cc:302] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.471966 25255 raft_consensus.cc:2781] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.472689 25255 raft_consensus.cc:686] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:17.436842 25281 raft_consensus.cc:2726] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.474001 25255 consensus_queue.cc:237] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:17.472731 25059 leader_election.cc:341] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.478811 25059 leader_election.cc:302] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.479872 25164 raft_consensus.cc:2454] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.485332 25318 raft_consensus.cc:2726] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.486140 25337 consensus_queue.cc:1035] T 70b9d51a2e094f0d8b45171217807a23 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.429308 25207 raft_consensus.cc:348] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.497321 25207 raft_consensus.cc:374] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.497660 25207 raft_consensus.cc:729] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.498976 25207 consensus_queue.cc:260] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.507668 25209 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.507895 25207 ts_tablet_manager.cc:1398] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.082s user 0.007s sys 0.001s
I20240501 13:56:17.508304 25209 ts_tablet_manager.cc:1367] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.204s user 0.037s sys 0.045s
I20240501 13:56:17.508970 25207 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.513900 25209 raft_consensus.cc:348] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.514853 25209 raft_consensus.cc:374] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.515301 25209 raft_consensus.cc:729] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.516101 25209 consensus_queue.cc:260] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.518154 25210 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.518631 25210 ts_tablet_manager.cc:1367] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.024s
I20240501 13:56:17.521124 25210 raft_consensus.cc:348] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.521854 25210 raft_consensus.cc:374] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.522126 25210 raft_consensus.cc:729] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.522743 25209 ts_tablet_manager.cc:1398] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.014s user 0.005s sys 0.002s
I20240501 13:56:17.522728 25210 consensus_queue.cc:260] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.523756 25209 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.523869 25018 raft_consensus.cc:2454] T 55e60d3e56d348209c02fb051fd3d27f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.524782 25210 ts_tablet_manager.cc:1398] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240501 13:56:17.524896 25059 leader_election.cc:302] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.525712 25210 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.526154 25318 raft_consensus.cc:2781] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.526701 25318 raft_consensus.cc:686] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:17.526885 24915 catalog_manager.cc:5484] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:17.527554 25318 consensus_queue.cc:237] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.529302 25207 tablet_bootstrap.cc:654] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.529995 25209 tablet_bootstrap.cc:654] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.529341 24915 catalog_manager.cc:5484] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:17.538806 25209 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.539284 25209 ts_tablet_manager.cc:1367] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20240501 13:56:17.542871 25209 raft_consensus.cc:348] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.543740 25209 raft_consensus.cc:374] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.545110 25210 tablet_bootstrap.cc:654] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.544104 25209 raft_consensus.cc:729] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.546217 25209 consensus_queue.cc:260] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.549414 25209 ts_tablet_manager.cc:1398] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20240501 13:56:17.550360 25209 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.559036 25209 tablet_bootstrap.cc:654] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.564239 25255 raft_consensus.cc:482] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.564841 25255 raft_consensus.cc:504] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.569978 25207 tablet_bootstrap.cc:492] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.570405 25210 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.570510 25207 ts_tablet_manager.cc:1367] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.006s
I20240501 13:56:17.570899 25210 ts_tablet_manager.cc:1367] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.017s
I20240501 13:56:17.572253 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.572897 25207 raft_consensus.cc:348] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.573604 25207 raft_consensus.cc:374] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.573369 25210 raft_consensus.cc:348] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.573881 25207 raft_consensus.cc:729] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.574029 25210 raft_consensus.cc:374] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.574359 25210 raft_consensus.cc:729] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.574548 25207 consensus_queue.cc:260] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.574952 25210 consensus_queue.cc:260] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.577049 25210 ts_tablet_manager.cc:1398] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240501 13:56:17.577909 25210 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.579352 25280 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.580289 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.581471 25255 leader_election.cc:288] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.582751 25207 ts_tablet_manager.cc:1398] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240501 13:56:17.583351 25210 tablet_bootstrap.cc:654] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.583706 25207 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
W20240501 13:56:17.583552 25058 leader_election.cc:341] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:17.586498 25059 leader_election.cc:341] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.586962 25059 leader_election.cc:302] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.588234 25280 raft_consensus.cc:2726] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.590183 25207 tablet_bootstrap.cc:654] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.596412 25280 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.600644 24915 catalog_manager.cc:5484] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:17.616032 25210 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.622241 25210 ts_tablet_manager.cc:1367] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.019s
I20240501 13:56:17.622227 25209 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.623178 25209 ts_tablet_manager.cc:1367] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.073s user 0.020s sys 0.008s
I20240501 13:56:17.623515 25337 raft_consensus.cc:482] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.624011 25337 raft_consensus.cc:504] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.626444 25209 raft_consensus.cc:348] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.627246 25209 raft_consensus.cc:374] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.627756 25209 raft_consensus.cc:729] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.628405 25209 consensus_queue.cc:260] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.629974 25209 ts_tablet_manager.cc:1398] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240501 13:56:17.630883 25209 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.626315 25210 raft_consensus.cc:348] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.632694 25210 raft_consensus.cc:374] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.632998 25210 raft_consensus.cc:729] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.633698 25210 consensus_queue.cc:260] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.641427 25210 ts_tablet_manager.cc:1398] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.018s user 0.003s sys 0.004s
I20240501 13:56:17.642235 25210 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.643812 25337 leader_election.cc:288] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.645692 25280 raft_consensus.cc:482] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.645761 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.646013 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.646549 25016 raft_consensus.cc:2454] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:17.646286 25280 raft_consensus.cc:504] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.648906 25280 leader_election.cc:288] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.646581 25090 raft_consensus.cc:2454] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:17.657099 25133 leader_election.cc:302] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.650949 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e50604381e4450b9f3c000fe9030fd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.657976 25164 raft_consensus.cc:2454] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:17.658233 25337 raft_consensus.cc:2781] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.658636 25337 raft_consensus.cc:482] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.658952 25337 raft_consensus.cc:3037] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.660337 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e50604381e4450b9f3c000fe9030fd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.661355 25058 leader_election.cc:302] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.662132 25280 raft_consensus.cc:2781] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 13:56:17.669149 25059 leader_election.cc:341] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.669521 25280 raft_consensus.cc:482] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.674252 25280 raft_consensus.cc:3037] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.682906 25337 raft_consensus.cc:504] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.687530 25207 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.687379 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.687956 25207 ts_tablet_manager.cc:1367] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.104s user 0.012s sys 0.000s
I20240501 13:56:17.687979 25018 raft_consensus.cc:3037] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.684033 25280 raft_consensus.cc:504] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.692386 25280 leader_election.cc:288] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.698531 25210 tablet_bootstrap.cc:654] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.690613 25207 raft_consensus.cc:348] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.699973 25207 raft_consensus.cc:374] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.700174 25207 raft_consensus.cc:729] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.700798 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e50604381e4450b9f3c000fe9030fd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:17.700718 25207 consensus_queue.cc:260] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.701385 25164 raft_consensus.cc:3037] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.702313 25207 ts_tablet_manager.cc:1398] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240501 13:56:17.703181 25207 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.703303 25209 tablet_bootstrap.cc:654] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.706838 25018 raft_consensus.cc:2454] T bebe230e116f4ecd84b6b251fa299925 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.708326 25133 leader_election.cc:302] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.701372 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e50604381e4450b9f3c000fe9030fd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.708959 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebe230e116f4ecd84b6b251fa299925" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:17.710522 25090 raft_consensus.cc:3037] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.710641 25269 raft_consensus.cc:482] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.711102 25269 raft_consensus.cc:504] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.713086 25269 leader_election.cc:288] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.713253 25294 raft_consensus.cc:2781] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.713743 25294 raft_consensus.cc:686] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:17.714463 25294 consensus_queue.cc:237] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.715155 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.715780 25093 raft_consensus.cc:2454] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:17.717128 25289 raft_consensus.cc:482] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.720333 25131 leader_election.cc:302] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.717621 25289 raft_consensus.cc:504] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.723872 25289 leader_election.cc:288] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.724503 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.725173 25017 raft_consensus.cc:2454] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
W20240501 13:56:17.726398 25059 leader_election.cc:341] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.721742 25269 raft_consensus.cc:2781] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.727445 25269 raft_consensus.cc:482] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:17.727950 25269 raft_consensus.cc:3037] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.728713 25133 leader_election.cc:302] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.728121 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:17.731922 25289 raft_consensus.cc:2781] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:17.732525 25289 raft_consensus.cc:482] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 13:56:17.732678 25131 leader_election.cc:341] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.733291 25289 raft_consensus.cc:3037] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.718582 25337 leader_election.cc:288] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.741252 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:17.744843 25207 tablet_bootstrap.cc:654] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:17.746279 25133 leader_election.cc:341] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.746574 25289 raft_consensus.cc:504] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.749971 25210 tablet_bootstrap.cc:492] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.750356 25210 ts_tablet_manager.cc:1367] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.030s
I20240501 13:56:17.753841 25090 raft_consensus.cc:2454] T bebe230e116f4ecd84b6b251fa299925 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.754472 25210 raft_consensus.cc:348] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.755232 25210 raft_consensus.cc:374] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.755517 25210 raft_consensus.cc:729] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.756145 25210 consensus_queue.cc:260] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.755905 25269 raft_consensus.cc:504] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.756328 24915 catalog_manager.cc:5484] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.758569 25210 ts_tablet_manager.cc:1398] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 13:56:17.760020 25164 raft_consensus.cc:2454] T b7e50604381e4450b9f3c000fe9030fd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:17.760936 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.771072 25269 leader_election.cc:288] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.771158 25018 raft_consensus.cc:3037] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.771435 25058 leader_election.cc:302] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:17.762573 25289 leader_election.cc:288] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:17.772172 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:17.768267 25209 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.772644 25280 raft_consensus.cc:2781] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.781149 25209 ts_tablet_manager.cc:1367] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.150s user 0.015s sys 0.001s
I20240501 13:56:17.781348 25210 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.781162 25280 raft_consensus.cc:686] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
W20240501 13:56:17.781781 25133 leader_election.cc:341] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.782430 25280 consensus_queue.cc:237] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.783648 25209 raft_consensus.cc:348] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.784520 25209 raft_consensus.cc:374] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.784795 25209 raft_consensus.cc:729] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.785892 25207 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.785542 25209 consensus_queue.cc:260] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.786352 25207 ts_tablet_manager.cc:1367] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.018s
I20240501 13:56:17.767163 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530c7a7742ff4dc182758c2250c1fabd" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:17.787070 25090 raft_consensus.cc:3037] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.787744 25209 ts_tablet_manager.cc:1398] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 13:56:17.788712 25209 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.789129 25210 tablet_bootstrap.cc:654] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.789222 25207 raft_consensus.cc:348] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.789978 25207 raft_consensus.cc:374] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.790282 25207 raft_consensus.cc:729] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.780267 25018 raft_consensus.cc:2454] T 530c7a7742ff4dc182758c2250c1fabd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.792091 25207 consensus_queue.cc:260] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.794761 25133 leader_election.cc:302] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:17.795306 25207 ts_tablet_manager.cc:1398] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 13:56:17.796304 25269 raft_consensus.cc:2781] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.798149 25209 tablet_bootstrap.cc:654] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.797523 25269 raft_consensus.cc:686] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:17.805223 25207 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.808547 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:17.809525 25088 raft_consensus.cc:3037] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.810374 25337 consensus_queue.cc:1035] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.809576 25090 raft_consensus.cc:2454] T 530c7a7742ff4dc182758c2250c1fabd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.806712 25269 consensus_queue.cc:237] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.822247 24913 catalog_manager.cc:5484] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:17.832044 25210 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.832571 25210 ts_tablet_manager.cc:1367] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.006s
I20240501 13:56:17.840474 25210 raft_consensus.cc:348] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.842916 25210 raft_consensus.cc:374] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.846995 25210 raft_consensus.cc:729] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.848919 25210 consensus_queue.cc:260] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.851011 25088 raft_consensus.cc:2454] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:17.851909 25210 ts_tablet_manager.cc:1398] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240501 13:56:17.852244 25131 leader_election.cc:302] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:17.852836 25210 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.853813 25269 raft_consensus.cc:2781] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:17.855996 25269 raft_consensus.cc:686] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:17.854652 24913 catalog_manager.cc:5484] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.859421 25209 tablet_bootstrap.cc:492] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.859951 25209 ts_tablet_manager.cc:1367] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.001s
I20240501 13:56:17.861614 25207 tablet_bootstrap.cc:654] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.866864 25209 raft_consensus.cc:348] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.867748 25209 raft_consensus.cc:374] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.868067 25209 raft_consensus.cc:729] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.871979 25209 consensus_queue.cc:260] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.874225 25209 ts_tablet_manager.cc:1398] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240501 13:56:17.875200 25209 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.871083 25210 tablet_bootstrap.cc:654] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:17.878964 25058 consensus_peers.cc:485] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e -> Peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Couldn't send request to peer ef020bad8d24499281f69de00e90af06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:17.865377 25269 consensus_queue.cc:237] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.882563 25214 consensus_queue.cc:1035] T 38ec4e2e661b4705afa2e4a2c4a76bd5 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.892212 25209 tablet_bootstrap.cc:654] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.893658 25207 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.894104 25207 ts_tablet_manager.cc:1367] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.089s user 0.016s sys 0.009s
I20240501 13:56:17.897732 25210 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:17.898480 25210 ts_tablet_manager.cc:1367] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.001s
I20240501 13:56:17.900597 25207 raft_consensus.cc:348] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.901371 25207 raft_consensus.cc:374] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.901648 25207 raft_consensus.cc:729] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.901240 24914 catalog_manager.cc:5484] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:17.902274 25207 consensus_queue.cc:260] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.902490 25210 raft_consensus.cc:348] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.904065 25210 raft_consensus.cc:374] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.904441 25210 raft_consensus.cc:729] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.905084 25210 consensus_queue.cc:260] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.907747 25210 ts_tablet_manager.cc:1398] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20240501 13:56:17.908082 25209 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:17.908597 25209 ts_tablet_manager.cc:1367] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.004s
I20240501 13:56:17.908682 25210 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:17.909289 25280 consensus_queue.cc:1035] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.911327 25209 raft_consensus.cc:348] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.912020 25209 raft_consensus.cc:374] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.912283 25209 raft_consensus.cc:729] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.912910 25209 consensus_queue.cc:260] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.914533 25210 tablet_bootstrap.cc:654] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.914865 25209 ts_tablet_manager.cc:1398] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 13:56:17.915760 25209 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:17.921841 25209 tablet_bootstrap.cc:654] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.926216 25207 ts_tablet_manager.cc:1398] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.032s user 0.002s sys 0.005s
I20240501 13:56:17.927546 25207 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:17.928900 25296 raft_consensus.cc:482] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.929605 25296 raft_consensus.cc:504] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.933836 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:17.940721 24982 leader_election.cc:341] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.946141 25207 tablet_bootstrap.cc:654] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:17.938766 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:17.950683 24983 leader_election.cc:341] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.951360 24983 leader_election.cc:302] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:17.953151 25211 raft_consensus.cc:2726] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:17.932312 25296 leader_election.cc:288] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.957459 25164 raft_consensus.cc:3037] T 55e60d3e56d348209c02fb051fd3d27f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:17.971122 25280 consensus_queue.cc:1035] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:17.974140 25207 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:17.975901 25207 ts_tablet_manager.cc:1367] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.009s
I20240501 13:56:17.978586 25207 raft_consensus.cc:348] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.979548 25207 raft_consensus.cc:374] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:17.980630 25280 consensus_queue.cc:1035] T 55e60d3e56d348209c02fb051fd3d27f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:17.986107 25255 raft_consensus.cc:482] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:17.986816 25255 raft_consensus.cc:504] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.991595 25255 leader_election.cc:288] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:17.993182 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:17.995783 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:17.995987 25058 leader_election.cc:341] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:17.979888 25207 raft_consensus.cc:729] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:17.997924 25207 consensus_queue.cc:260] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:17.999754 25207 ts_tablet_manager.cc:1398] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
W20240501 13:56:18.000478 25059 leader_election.cc:341] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.001066 25059 leader_election.cc:302] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.001241 25207 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.001956 25255 raft_consensus.cc:2726] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.007061 25296 raft_consensus.cc:482] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.007942 25296 raft_consensus.cc:504] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.009320 25211 raft_consensus.cc:482] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.009805 25211 raft_consensus.cc:504] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.011339 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a49139fcaa8462593981602f13840c4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:18.012773 24983 leader_election.cc:341] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.017642 25207 tablet_bootstrap.cc:654] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.017946 25369 raft_consensus.cc:482] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.018426 25369 raft_consensus.cc:504] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.019042 25318 consensus_queue.cc:1035] T 357df45144824ef386e687e270aa8e77 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:18.023046 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.024984 25318 raft_consensus.cc:482] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.025717 25211 leader_election.cc:288] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.028895 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.029570 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.025499 25318 raft_consensus.cc:504] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.026520 25369 leader_election.cc:288] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.028429 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.031148 25166 raft_consensus.cc:2454] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
W20240501 13:56:18.028156 24982 leader_election.cc:341] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:18.032208 24983 leader_election.cc:341] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:18.032763 24982 leader_election.cc:341] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.032848 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.033102 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a49139fcaa8462593981602f13840c4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.033237 24982 leader_election.cc:302] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.034014 25369 raft_consensus.cc:2726] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.033773 25088 raft_consensus.cc:2454] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:18.032114 25318 leader_election.cc:288] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:18.034703 25058 leader_election.cc:341] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.035706 24982 leader_election.cc:302] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.035706 24983 leader_election.cc:302] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:18.037209 25296 leader_election.cc:288] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.037508 25409 raft_consensus.cc:2781] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.038103 25409 raft_consensus.cc:482] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.038515 25409 raft_consensus.cc:3037] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.037478 25369 raft_consensus.cc:2781] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.041651 25369 raft_consensus.cc:482] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.042141 25369 raft_consensus.cc:3037] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.040846 25207 tablet_bootstrap.cc:492] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.045939 25207 ts_tablet_manager.cc:1367] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.045s user 0.015s sys 0.011s
I20240501 13:56:18.051770 25015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:18.075135 25059 leader_election.cc:341] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.082486 25318 consensus_queue.cc:1035] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20240501 13:56:18.075688 25059 leader_election.cc:302] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.129047 25318 raft_consensus.cc:2726] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 13:56:18.129223 25059 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.129:33309, user_credentials={real_user=slave}} blocked reactor thread for 59924.3us
I20240501 13:56:18.131340 25409 raft_consensus.cc:504] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.129841 25369 raft_consensus.cc:504] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.134631 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a49139fcaa8462593981602f13840c4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:18.135236 25093 raft_consensus.cc:3037] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.136834 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
W20240501 13:56:18.138852 24982 leader_election.cc:341] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.143805 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a49139fcaa8462593981602f13840c4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:18.144472 25209 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.144918 25209 ts_tablet_manager.cc:1367] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.229s user 0.087s sys 0.110s
W20240501 13:56:18.145227 24983 leader_election.cc:341] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.145663 25210 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.145805 25369 leader_election.cc:288] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.146096 25210 ts_tablet_manager.cc:1367] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.238s user 0.018s sys 0.047s
I20240501 13:56:18.147529 25093 raft_consensus.cc:2454] T 8a49139fcaa8462593981602f13840c4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:18.148638 24982 leader_election.cc:302] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.149705 25369 raft_consensus.cc:2781] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.151243 25409 leader_election.cc:288] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.153273 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:18.153889 25162 raft_consensus.cc:3037] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.163311 25162 raft_consensus.cc:2454] T c1954780552c46cb829f499d56d225c7 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:18.164273 24983 leader_election.cc:302] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:18.165439 25409 raft_consensus.cc:2781] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.165284 25210 raft_consensus.cc:348] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.166218 25409 raft_consensus.cc:686] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:18.165189 25209 raft_consensus.cc:348] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.166766 25209 raft_consensus.cc:374] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.166390 25210 raft_consensus.cc:374] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.166960 25207 raft_consensus.cc:348] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.167112 25209 raft_consensus.cc:729] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.150417 25369 raft_consensus.cc:686] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:18.167917 25207 raft_consensus.cc:374] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.168370 25209 consensus_queue.cc:260] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.168735 25369 consensus_queue.cc:237] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.167102 25409 consensus_queue.cc:237] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.167649 25210 raft_consensus.cc:729] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.177764 25209 ts_tablet_manager.cc:1398] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.033s user 0.005s sys 0.003s
I20240501 13:56:18.178766 25209 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.168294 25207 raft_consensus.cc:729] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.177742 25210 consensus_queue.cc:260] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.183018 25207 consensus_queue.cc:260] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.187355 25209 tablet_bootstrap.cc:654] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.189823 25207 ts_tablet_manager.cc:1398] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.143s user 0.010s sys 0.012s
I20240501 13:56:18.190330 25210 ts_tablet_manager.cc:1398] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.044s user 0.006s sys 0.002s
I20240501 13:56:18.190829 25207 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.191260 25210 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:18.194808 24914 catalog_manager.cc:5484] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:18.200676 25210 tablet_bootstrap.cc:654] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:18.203918 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:18.212198 25289 raft_consensus.cc:482] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.212983 25289 raft_consensus.cc:504] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.235044 25214 raft_consensus.cc:482] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.235608 25214 raft_consensus.cc:504] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.271095 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:18.273291 25133 leader_election.cc:341] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.279314 25016 raft_consensus.cc:3037] T b7e50604381e4450b9f3c000fe9030fd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.282208 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:18.284392 25133 leader_election.cc:341] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.290129 25318 consensus_queue.cc:1035] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.298070 25289 leader_election.cc:288] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.301066 25214 leader_election.cc:288] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.302978 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.304101 25093 raft_consensus.cc:2454] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:18.305327 25289 consensus_queue.cc:1035] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.308145 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.310424 25131 leader_election.cc:302] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.312507 25269 raft_consensus.cc:2781] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.313097 25269 raft_consensus.cc:482] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.313550 25269 raft_consensus.cc:3037] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.317867 25296 raft_consensus.cc:482] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.318562 25296 raft_consensus.cc:504] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.326435 25269 raft_consensus.cc:504] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.327383 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.327701 25296 leader_election.cc:288] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:18.329533 24983 leader_election.cc:341] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:18.332258 25131 leader_election.cc:341] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.332988 25131 leader_election.cc:302] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.335269 25214 raft_consensus.cc:2726] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.344818 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.345204 25217 consensus_queue.cc:1035] T b7e50604381e4450b9f3c000fe9030fd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.345844 25093 raft_consensus.cc:2454] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
W20240501 13:56:18.355870 25133 consensus_peers.cc:485] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 -> Peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Couldn't send request to peer 346b2bb141744d459f1fa3ad602d1a15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:18.368176 24982 leader_election.cc:302] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.369601 25409 raft_consensus.cc:2781] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.370079 25409 raft_consensus.cc:482] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.370460 25409 raft_consensus.cc:3037] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.370874 25289 consensus_queue.cc:1035] T bebe230e116f4ecd84b6b251fa299925 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.371822 25214 consensus_queue.cc:1035] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:18.377718 25209 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.378186 25209 ts_tablet_manager.cc:1367] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.200s user 0.021s sys 0.059s
I20240501 13:56:18.378515 25409 raft_consensus.cc:504] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.380998 25409 leader_election.cc:288] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.381623 25269 leader_election.cc:288] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.382419 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:18.383055 25207 tablet_bootstrap.cc:654] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.382974 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
W20240501 13:56:18.384738 24983 leader_election.cc:341] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:18.385782 25133 leader_election.cc:341] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.386523 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:18.387123 25088 raft_consensus.cc:3037] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.387892 25269 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:18.389197 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:18.389801 25091 raft_consensus.cc:3037] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.395196 25210 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.395838 25091 raft_consensus.cc:2454] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:18.398133 25210 ts_tablet_manager.cc:1367] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.207s user 0.035s sys 0.047s
I20240501 13:56:18.399159 25131 leader_election.cc:302] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.397619 24914 catalog_manager.cc:5484] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:18.395789 25088 raft_consensus.cc:2454] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:18.404155 25214 raft_consensus.cc:2781] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.404731 24982 leader_election.cc:302] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.405699 25214 raft_consensus.cc:686] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:18.405808 25409 raft_consensus.cc:2781] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.406344 25409 raft_consensus.cc:686] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:18.406587 25210 raft_consensus.cc:348] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.408138 25207 tablet_bootstrap.cc:492] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.409324 25207 ts_tablet_manager.cc:1367] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.000s
I20240501 13:56:18.407598 25409 consensus_queue.cc:237] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.408921 25209 raft_consensus.cc:348] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.412705 25209 raft_consensus.cc:374] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.413054 25209 raft_consensus.cc:729] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.413338 25318 raft_consensus.cc:482] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.413820 25318 raft_consensus.cc:504] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.413964 25209 consensus_queue.cc:260] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.408943 25210 raft_consensus.cc:374] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.414778 25210 raft_consensus.cc:729] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.406663 25214 consensus_queue.cc:237] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.418838 25210 consensus_queue.cc:260] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.423275 25210 ts_tablet_manager.cc:1398] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20240501 13:56:18.416127 25209 ts_tablet_manager.cc:1398] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20240501 13:56:18.422086 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.426354 25209 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.426440 25210 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:18.428092 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.428823 25017 raft_consensus.cc:2454] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
W20240501 13:56:18.426805 25058 leader_election.cc:341] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.430636 25059 leader_election.cc:302] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.431550 25217 raft_consensus.cc:2781] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.431914 25217 raft_consensus.cc:482] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.432257 25217 raft_consensus.cc:3037] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.432957 25209 tablet_bootstrap.cc:654] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.424181 25207 raft_consensus.cc:348] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.434566 25207 raft_consensus.cc:374] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.434880 25207 raft_consensus.cc:729] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.415807 25318 leader_election.cc:288] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.436398 25207 consensus_queue.cc:260] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.440829 25207 ts_tablet_manager.cc:1398] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20240501 13:56:18.441776 25207 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.443593 25217 raft_consensus.cc:504] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.445626 25294 consensus_queue.cc:1035] T 530c7a7742ff4dc182758c2250c1fabd P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.440940 25210 tablet_bootstrap.cc:654] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.446561 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:18.447181 25017 raft_consensus.cc:3037] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.452983 25207 tablet_bootstrap.cc:654] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.471596 25217 leader_election.cc:288] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.476557 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:18.481222 25210 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.481905 25210 ts_tablet_manager.cc:1367] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.003s
W20240501 13:56:18.487105 25058 leader_election.cc:341] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.493577 25269 raft_consensus.cc:482] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.494086 25269 raft_consensus.cc:504] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.496057 25269 leader_election.cc:288] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.491700 24913 catalog_manager.cc:5484] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:18.491441 25210 raft_consensus.cc:348] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.502241 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.504727 25210 raft_consensus.cc:374] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.505196 25013 raft_consensus.cc:2454] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:18.499051 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.505568 25210 raft_consensus.cc:729] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.508507 25133 leader_election.cc:302] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:18.510646 25090 raft_consensus.cc:2454] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:18.512374 25210 consensus_queue.cc:260] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.515069 25210 ts_tablet_manager.cc:1398] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.033s user 0.006s sys 0.002s
I20240501 13:56:18.513998 25209 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.516553 25209 ts_tablet_manager.cc:1367] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.090s user 0.018s sys 0.005s
I20240501 13:56:18.517407 25210 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:18.522670 25017 raft_consensus.cc:2454] T b6ae8ea0d9e240c29296a5b826f67196 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:18.520401 25209 raft_consensus.cc:348] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.526237 25209 raft_consensus.cc:374] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.526489 25209 raft_consensus.cc:729] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.527015 25209 consensus_queue.cc:260] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.512622 25214 raft_consensus.cc:2781] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.530141 24914 catalog_manager.cc:5484] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:18.540241 25209 ts_tablet_manager.cc:1398] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240501 13:56:18.536933 25214 raft_consensus.cc:482] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.544209 25209 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.549082 25059 leader_election.cc:302] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.550439 25207 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.576022 25207 ts_tablet_manager.cc:1367] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.134s user 0.022s sys 0.012s
I20240501 13:56:18.578838 25207 raft_consensus.cc:348] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.579612 25207 raft_consensus.cc:374] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.579898 25207 raft_consensus.cc:729] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.580570 25207 consensus_queue.cc:260] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.584723 25207 ts_tablet_manager.cc:1398] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240501 13:56:18.551029 25318 raft_consensus.cc:2781] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.585690 25207 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.585649 25318 raft_consensus.cc:686] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:18.544126 25214 raft_consensus.cc:3037] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.586599 25318 consensus_queue.cc:237] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.595479 25210 tablet_bootstrap.cc:654] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.600692 25209 tablet_bootstrap.cc:654] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.604086 25207 tablet_bootstrap.cc:654] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.607221 25210 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.607758 25210 ts_tablet_manager.cc:1367] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.001s
I20240501 13:56:18.607575 25214 raft_consensus.cc:504] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.610929 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:18.611368 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e985a77279a46ec98e6d3164f7756c2" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:18.611562 25017 raft_consensus.cc:3037] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.611932 25092 raft_consensus.cc:3037] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.618845 25017 raft_consensus.cc:2454] T 8e985a77279a46ec98e6d3164f7756c2 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:18.621102 25092 raft_consensus.cc:2454] T 8e985a77279a46ec98e6d3164f7756c2 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:18.621021 25409 consensus_queue.cc:1035] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:18.621803 25133 leader_election.cc:302] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:18.622479 25214 leader_election.cc:288] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.622718 25269 raft_consensus.cc:2781] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.623216 25269 raft_consensus.cc:686] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:18.624068 25269 consensus_queue.cc:237] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.620326 25210 raft_consensus.cc:348] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.633037 25210 raft_consensus.cc:374] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.633440 25210 raft_consensus.cc:729] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.634202 25210 consensus_queue.cc:260] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.636651 25210 ts_tablet_manager.cc:1398] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.029s user 0.003s sys 0.003s
I20240501 13:56:18.637868 25210 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:18.647099 25207 tablet_bootstrap.cc:492] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.647267 25210 tablet_bootstrap.cc:654] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.647615 25207 ts_tablet_manager.cc:1367] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.008s
I20240501 13:56:18.654273 25409 raft_consensus.cc:482] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.659766 25409 raft_consensus.cc:504] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.660619 24913 catalog_manager.cc:5484] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:18.661547 25210 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.661814 25409 leader_election.cc:288] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.661983 25210 ts_tablet_manager.cc:1367] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.008s
I20240501 13:56:18.664716 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:18.666857 24983 leader_election.cc:341] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.672057 25210 raft_consensus.cc:348] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.672832 25210 raft_consensus.cc:374] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.677367 25210 raft_consensus.cc:729] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.680193 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.680836 25210 consensus_queue.cc:260] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.682453 25092 raft_consensus.cc:2454] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:18.683996 25210 ts_tablet_manager.cc:1398] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20240501 13:56:18.684940 25210 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
W20240501 13:56:18.679944 24983 consensus_peers.cc:485] T 8a49139fcaa8462593981602f13840c4 P 346b2bb141744d459f1fa3ad602d1a15 -> Peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Couldn't send request to peer ef020bad8d24499281f69de00e90af06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240501 13:56:18.678802 24982 consensus_peers.cc:485] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 -> Peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Couldn't send request to peer 171c5cc5f6df4820bc16946f8717737e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:18.679188 25207 raft_consensus.cc:348] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.687582 25207 raft_consensus.cc:374] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.687855 25207 raft_consensus.cc:729] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.687974 24982 leader_election.cc:302] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.688757 25207 consensus_queue.cc:260] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.689363 25409 raft_consensus.cc:2781] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.689796 25409 raft_consensus.cc:482] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.690227 25409 raft_consensus.cc:3037] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.691195 25207 ts_tablet_manager.cc:1398] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.043s user 0.004s sys 0.004s
I20240501 13:56:18.691700 25210 tablet_bootstrap.cc:654] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.692128 25207 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.698118 25207 tablet_bootstrap.cc:654] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.678304 25209 tablet_bootstrap.cc:492] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.698160 25409 raft_consensus.cc:504] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.700578 25209 ts_tablet_manager.cc:1367] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.157s user 0.012s sys 0.023s
I20240501 13:56:18.703694 25209 raft_consensus.cc:348] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.706913 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:18.708659 25092 raft_consensus.cc:3037] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.708804 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
W20240501 13:56:18.711802 24983 leader_election.cc:341] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.707365 24916 catalog_manager.cc:5484] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:18.716675 25209 raft_consensus.cc:374] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.707808 25409 leader_election.cc:288] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.717259 25209 raft_consensus.cc:729] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.718675 25209 consensus_queue.cc:260] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.725750 25207 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.725926 25209 ts_tablet_manager.cc:1398] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240501 13:56:18.726238 25207 ts_tablet_manager.cc:1367] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.004s
I20240501 13:56:18.728523 25209 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.730199 25207 raft_consensus.cc:348] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.730927 25207 raft_consensus.cc:374] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.731201 25207 raft_consensus.cc:729] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.731803 25207 consensus_queue.cc:260] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.727273 25092 raft_consensus.cc:2454] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:18.733683 25209 tablet_bootstrap.cc:654] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.733865 25207 ts_tablet_manager.cc:1398] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240501 13:56:18.735239 25207 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.736902 25409 raft_consensus.cc:482] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.737515 25409 raft_consensus.cc:504] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.740609 25409 leader_election.cc:288] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.742763 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.743615 25090 raft_consensus.cc:2379] T d1d28bb928464ca7b989d4597a5b1d9c P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:18.745076 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d28bb928464ca7b989d4597a5b1d9c" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.745926 24982 leader_election.cc:302] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.746333 25269 raft_consensus.cc:482] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.746812 25269 raft_consensus.cc:504] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.747237 25409 raft_consensus.cc:2781] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:18.748153 25409 raft_consensus.cc:686] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:18.749131 25269 leader_election.cc:288] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.749348 25409 consensus_queue.cc:237] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.751025 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.750989 24983 leader_election.cc:302] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.752755 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7e26b87b054270bc41f6e57ff69ac0" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.753252 25369 raft_consensus.cc:3037] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.753638 25090 raft_consensus.cc:2379] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:18.756801 25133 leader_election.cc:302] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.757493 25207 tablet_bootstrap.cc:654] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.759526 25318 raft_consensus.cc:482] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.759725 25414 raft_consensus.cc:482] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.760035 25318 raft_consensus.cc:504] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.760152 25414 raft_consensus.cc:504] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.761976 25318 leader_election.cc:288] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.765197 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.765385 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.765018 25269 raft_consensus.cc:3037] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.814791 25296 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
W20240501 13:56:18.830127 25059 leader_election.cc:341] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:18.830312 25058 leader_election.cc:341] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.765379 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:18.831835 25059 leader_election.cc:341] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.832330 25059 leader_election.cc:302] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:18.833369 25318 raft_consensus.cc:2726] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.833968 25207 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:18.834373 25209 tablet_bootstrap.cc:492] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.761988 25414 leader_election.cc:288] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.834997 25209 ts_tablet_manager.cc:1367] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.107s user 0.022s sys 0.034s
I20240501 13:56:18.778764 25369 raft_consensus.cc:2726] T d1d28bb928464ca7b989d4597a5b1d9c P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.822383 24914 catalog_manager.cc:5484] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:18.838605 25209 raft_consensus.cc:348] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.839356 25209 raft_consensus.cc:374] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.839723 25209 raft_consensus.cc:729] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.834426 25207 ts_tablet_manager.cc:1367] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.003s
I20240501 13:56:18.840517 25269 raft_consensus.cc:2726] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.842950 25207 raft_consensus.cc:348] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.843822 25207 raft_consensus.cc:374] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.844177 25207 raft_consensus.cc:729] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.786131 25409 consensus_queue.cc:1035] T c1954780552c46cb829f499d56d225c7 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.059s
I20240501 13:56:18.846666 25269 raft_consensus.cc:482] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.845008 25207 consensus_queue.cc:260] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.793807 25210 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:18.848002 25210 ts_tablet_manager.cc:1367] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.163s user 0.030s sys 0.009s
I20240501 13:56:18.848695 25207 ts_tablet_manager.cc:1398] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 13:56:18.849594 25207 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:18.765208 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.850400 25162 raft_consensus.cc:2454] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:18.847147 25269 raft_consensus.cc:504] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.851302 25058 leader_election.cc:302] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.840543 25209 consensus_queue.cc:260] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.853324 25209 ts_tablet_manager.cc:1398] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240501 13:56:18.853797 25414 raft_consensus.cc:2781] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.854171 25414 raft_consensus.cc:482] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.854480 25209 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.854535 25414 raft_consensus.cc:3037] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.859212 25210 raft_consensus.cc:348] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.860689 25209 tablet_bootstrap.cc:654] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.861331 25409 raft_consensus.cc:482] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.861831 25409 raft_consensus.cc:504] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.863168 25414 raft_consensus.cc:504] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.866027 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:18.867225 25414 leader_election.cc:288] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:18.867419 25059 leader_election.cc:341] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.868753 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:18.869339 25162 raft_consensus.cc:3037] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.873457 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.877075 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "357df45144824ef386e687e270aa8e77" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.878892 25209 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.879437 25209 ts_tablet_manager.cc:1367] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.010s
I20240501 13:56:18.881968 25209 raft_consensus.cc:348] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.882735 25209 raft_consensus.cc:374] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.883139 25209 raft_consensus.cc:729] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.883754 25209 consensus_queue.cc:260] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.879575 25133 leader_election.cc:302] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:18.886024 25289 raft_consensus.cc:2726] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:18.885246 25269 leader_election.cc:288] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:18.885448 25209 ts_tablet_manager.cc:1398] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240501 13:56:18.890923 25209 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.897845 25209 tablet_bootstrap.cc:654] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.905870 25210 raft_consensus.cc:374] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.906270 25210 raft_consensus.cc:729] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.906922 25210 consensus_queue.cc:260] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.910259 25296 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:18.913353 25209 tablet_bootstrap.cc:492] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:18.913904 25209 ts_tablet_manager.cc:1367] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.004s
I20240501 13:56:18.915148 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f1ef588c2a42b4b724543c340ab182" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:18.918627 25209 raft_consensus.cc:348] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.919392 25209 raft_consensus.cc:374] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:18.919664 25209 raft_consensus.cc:729] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:18.921759 25409 leader_election.cc:288] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.922556 25209 consensus_queue.cc:260] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.924225 25210 ts_tablet_manager.cc:1398] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.076s user 0.006s sys 0.000s
I20240501 13:56:18.925640 25210 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:18.931504 25207 tablet_bootstrap.cc:654] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.930505 25164 raft_consensus.cc:2454] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:18.935861 24983 leader_election.cc:302] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:18.934296 25209 ts_tablet_manager.cc:1398] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.020s user 0.003s sys 0.007s
I20240501 13:56:18.937392 25209 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:18.942734 25409 raft_consensus.cc:2781] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 13:56:18.944262 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 51538.8us
I20240501 13:56:18.947958 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f1ef588c2a42b4b724543c340ab182" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:18.949131 25092 raft_consensus.cc:2454] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:18.945082 25409 raft_consensus.cc:482] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.950505 25409 raft_consensus.cc:3037] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.951898 25209 tablet_bootstrap.cc:654] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.974433 25210 tablet_bootstrap.cc:654] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:18.976256 25414 raft_consensus.cc:482] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:18.977008 25414 raft_consensus.cc:504] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:18.980907 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:18.981616 25017 raft_consensus.cc:2454] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:18.983126 25059 leader_election.cc:302] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:18.983987 25217 raft_consensus.cc:2781] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:18.984367 25217 raft_consensus.cc:482] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:18.984686 25217 raft_consensus.cc:3037] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:18.987825 25414 leader_election.cc:288] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:18.989005 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:18.996399 25058 leader_election.cc:341] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:18.999562 25162 raft_consensus.cc:2454] T 17458082ba194fc8b534019c9eab0485 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:19.003319 25207 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.004722 25207 ts_tablet_manager.cc:1367] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.155s user 0.018s sys 0.007s
I20240501 13:56:19.007395 25058 leader_election.cc:302] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:19.008512 25414 raft_consensus.cc:2781] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.009441 25414 raft_consensus.cc:686] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:19.009263 25409 raft_consensus.cc:504] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.010529 25414 consensus_queue.cc:237] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.012123 25209 tablet_bootstrap.cc:492] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.012607 25209 ts_tablet_manager.cc:1367] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.004s
I20240501 13:56:19.015174 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f1ef588c2a42b4b724543c340ab182" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:19.015825 25162 raft_consensus.cc:3037] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.015596 25209 raft_consensus.cc:348] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.016319 25209 raft_consensus.cc:374] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.016644 25209 raft_consensus.cc:729] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
W20240501 13:56:19.017767 25058 consensus_peers.cc:485] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e -> Peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Couldn't send request to peer ef020bad8d24499281f69de00e90af06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:19.015214 25217 raft_consensus.cc:504] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.017722 25209 consensus_queue.cc:260] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.019361 25289 consensus_queue.cc:1035] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.032025 25209 ts_tablet_manager.cc:1398] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.019s user 0.005s sys 0.006s
I20240501 13:56:19.032974 25209 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.034803 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:19.035640 25017 raft_consensus.cc:3037] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.037712 25207 raft_consensus.cc:348] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.038347 25209 tablet_bootstrap.cc:654] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.038470 25207 raft_consensus.cc:374] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.038802 25207 raft_consensus.cc:729] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.044631 25217 leader_election.cc:288] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.044744 25207 consensus_queue.cc:260] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.045289 25409 leader_election.cc:288] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.051486 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
W20240501 13:56:19.052888 25058 leader_election.cc:341] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.051275 25294 consensus_queue.cc:1035] T d1d28bb928464ca7b989d4597a5b1d9c P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.055212 25207 ts_tablet_manager.cc:1398] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.050s user 0.002s sys 0.005s
I20240501 13:56:19.056583 25207 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.055397 25449 raft_consensus.cc:482] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.060259 25449 raft_consensus.cc:504] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.060796 25162 raft_consensus.cc:2454] T 72f1ef588c2a42b4b724543c340ab182 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.062906 24983 leader_election.cc:302] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:19.063653 25449 leader_election.cc:288] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.063987 25450 raft_consensus.cc:2781] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.066291 25450 raft_consensus.cc:686] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:19.068817 25017 raft_consensus.cc:2454] T 4e912ea2c2074a20b15c82376861b3f7 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:19.068490 25450 consensus_queue.cc:237] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.071213 25059 leader_election.cc:302] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.071264 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.073372 25217 raft_consensus.cc:2781] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.073880 25217 raft_consensus.cc:686] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:19.074736 25217 consensus_queue.cc:237] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.089080 25210 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.089706 25210 ts_tablet_manager.cc:1367] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.164s user 0.024s sys 0.012s
I20240501 13:56:19.090747 25207 tablet_bootstrap.cc:654] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.092510 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14ca53436a04447b62816101d8b838b" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.093469 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f1ef588c2a42b4b724543c340ab182" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:19.094147 25092 raft_consensus.cc:3037] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.098984 24982 leader_election.cc:302] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.100129 25450 raft_consensus.cc:2726] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.101155 25209 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.104720 25209 ts_tablet_manager.cc:1367] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.004s
I20240501 13:56:19.103062 25092 raft_consensus.cc:2454] T 72f1ef588c2a42b4b724543c340ab182 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.108924 25209 raft_consensus.cc:348] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.109661 25209 raft_consensus.cc:374] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.110875 25209 raft_consensus.cc:729] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.111604 25209 consensus_queue.cc:260] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.115605 25162 raft_consensus.cc:3037] T 357df45144824ef386e687e270aa8e77 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.118847 25209 ts_tablet_manager.cc:1398] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20240501 13:56:19.119369 24916 catalog_manager.cc:5484] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:19.121181 25209 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.127341 25209 tablet_bootstrap.cc:654] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.126940 24914 catalog_manager.cc:5484] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:19.131073 25269 raft_consensus.cc:482] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.131582 25269 raft_consensus.cc:504] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.131426 24916 catalog_manager.cc:5484] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:19.132089 25207 tablet_bootstrap.cc:492] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.132813 25207 ts_tablet_manager.cc:1367] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.002s
I20240501 13:56:19.133673 25269 leader_election.cc:288] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.135586 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.135641 25207 raft_consensus.cc:348] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.136598 25207 raft_consensus.cc:374] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.136938 25207 raft_consensus.cc:729] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.137813 25207 consensus_queue.cc:260] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.140669 25210 raft_consensus.cc:348] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.141470 25210 raft_consensus.cc:374] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.141741 25210 raft_consensus.cc:729] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.142416 25210 consensus_queue.cc:260] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.144515 25210 ts_tablet_manager.cc:1398] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.055s user 0.005s sys 0.003s
I20240501 13:56:19.144990 25414 consensus_queue.cc:1035] T b6ae8ea0d9e240c29296a5b826f67196 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.145426 25210 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.146423 25209 tablet_bootstrap.cc:492] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.146901 25209 ts_tablet_manager.cc:1367] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.002s
I20240501 13:56:19.153507 25207 ts_tablet_manager.cc:1398] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 13:56:19.150810 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1fd409e4d541dca63ed7a0544b4c5e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.154515 25207 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.158864 25209 raft_consensus.cc:348] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.159870 25209 raft_consensus.cc:374] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.160233 25209 raft_consensus.cc:729] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.168727 25090 raft_consensus.cc:2379] T aa1fd409e4d541dca63ed7a0544b4c5e P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:19.170011 25131 leader_election.cc:302] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:19.171304 25269 raft_consensus.cc:3037] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.174643 25209 consensus_queue.cc:260] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.180863 25209 ts_tablet_manager.cc:1398] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.034s user 0.005s sys 0.005s
I20240501 13:56:19.181555 25209 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.190167 25210 tablet_bootstrap.cc:654] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.197659 25269 raft_consensus.cc:2726] T aa1fd409e4d541dca63ed7a0544b4c5e P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.200507 25294 consensus_queue.cc:1035] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.207181 25207 tablet_bootstrap.cc:654] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.216326 25450 consensus_queue.cc:1035] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.218204 25409 consensus_queue.cc:1035] T aa1fd409e4d541dca63ed7a0544b4c5e P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.221944 25269 consensus_queue.cc:1035] T 8e985a77279a46ec98e6d3164f7756c2 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.224891 25209 tablet_bootstrap.cc:654] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.246618 25294 raft_consensus.cc:482] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.256633 25294 raft_consensus.cc:504] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.259965 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.262099 25294 leader_election.cc:288] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.264182 25210 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.264691 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ae8ea0d9e240c29296a5b826f67196" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.277289 25210 ts_tablet_manager.cc:1367] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.010s
I20240501 13:56:19.278810 25207 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.279294 25207 ts_tablet_manager.cc:1367] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.003s
I20240501 13:56:19.280537 25131 leader_election.cc:302] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:19.281649 25294 raft_consensus.cc:2726] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.279779 25210 raft_consensus.cc:348] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.282119 25210 raft_consensus.cc:374] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.282222 25207 raft_consensus.cc:348] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.282907 25207 raft_consensus.cc:374] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.283186 25207 raft_consensus.cc:729] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.283790 25207 consensus_queue.cc:260] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.286612 25207 ts_tablet_manager.cc:1398] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240501 13:56:19.288869 25209 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.289327 25209 ts_tablet_manager.cc:1367] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.108s user 0.009s sys 0.002s
I20240501 13:56:19.294596 25166 raft_consensus.cc:3037] T 8a49139fcaa8462593981602f13840c4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.282434 25210 raft_consensus.cc:729] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.287489 25207 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.296694 25209 raft_consensus.cc:348] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.297731 25209 raft_consensus.cc:374] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.298017 25209 raft_consensus.cc:729] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.298619 25209 consensus_queue.cc:260] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.301007 25209 ts_tablet_manager.cc:1398] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240501 13:56:19.300626 25210 consensus_queue.cc:260] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.301911 25209 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.304113 25210 ts_tablet_manager.cc:1398] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.026s user 0.004s sys 0.002s
I20240501 13:56:19.306540 25210 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.328166 25210 tablet_bootstrap.cc:654] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.330009 25207 tablet_bootstrap.cc:654] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.334262 25209 tablet_bootstrap.cc:654] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.343451 25289 raft_consensus.cc:482] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.344125 25289 raft_consensus.cc:504] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.347074 25409 raft_consensus.cc:482] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.347633 25289 leader_election.cc:288] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.347783 25409 raft_consensus.cc:504] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.352053 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.353638 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:19.354372 25131 leader_election.cc:341] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.354575 25090 raft_consensus.cc:2454] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:19.357368 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.358379 25210 tablet_bootstrap.cc:492] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.358321 25016 raft_consensus.cc:2454] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:19.359103 25210 ts_tablet_manager.cc:1367] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.005s
I20240501 13:56:19.360497 25133 leader_election.cc:302] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:19.362023 25289 raft_consensus.cc:2781] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:19.362535 25289 raft_consensus.cc:482] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:19.362975 25289 raft_consensus.cc:3037] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.363972 25210 raft_consensus.cc:348] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.365209 25210 raft_consensus.cc:374] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.367012 25209 tablet_bootstrap.cc:492] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.367444 25209 ts_tablet_manager.cc:1367] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.066s user 0.019s sys 0.000s
I20240501 13:56:19.365537 25210 raft_consensus.cc:729] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.368517 25207 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.369063 25207 ts_tablet_manager.cc:1367] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.025s
I20240501 13:56:19.369128 25210 consensus_queue.cc:260] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.372450 25210 ts_tablet_manager.cc:1398] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240501 13:56:19.371886 25207 raft_consensus.cc:348] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.372910 25207 raft_consensus.cc:374] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.373214 25207 raft_consensus.cc:729] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.373564 25210 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.373824 25207 consensus_queue.cc:260] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.369418 24982 leader_election.cc:302] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:19.376295 25450 raft_consensus.cc:2781] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:19.370707 25209 raft_consensus.cc:348] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.378142 25209 raft_consensus.cc:374] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.378376 25209 raft_consensus.cc:729] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.376675 25450 raft_consensus.cc:482] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:19.379695 25214 raft_consensus.cc:482] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.379842 25450 raft_consensus.cc:3037] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.380199 25214 raft_consensus.cc:504] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.382185 25214 leader_election.cc:288] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.382378 25289 raft_consensus.cc:504] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.382618 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.383311 25166 raft_consensus.cc:2454] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:19.383486 25409 leader_election.cc:288] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.383635 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e059d8fd99574a5eac0b49e9405d7bd4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.384737 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e059d8fd99574a5eac0b49e9405d7bd4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.384989 25207 ts_tablet_manager.cc:1398] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20240501 13:56:19.385361 25016 raft_consensus.cc:2454] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:19.384948 25209 consensus_queue.cc:260] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.386063 25207 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.387284 25133 leader_election.cc:302] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
W20240501 13:56:19.388015 25131 leader_election.cc:341] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.391369 25450 raft_consensus.cc:504] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.392112 25269 raft_consensus.cc:2781] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:19.392536 25269 raft_consensus.cc:482] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:19.392858 25269 raft_consensus.cc:3037] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.404062 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:19.404783 25016 raft_consensus.cc:3037] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.407068 25289 leader_election.cc:288] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.406877 25210 tablet_bootstrap.cc:654] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.409727 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:19.410413 25166 raft_consensus.cc:3037] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:19.411875 25059 consensus_peers.cc:485] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e -> Peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Couldn't send request to peer 346b2bb141744d459f1fa3ad602d1a15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:19.413244 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dff22b19a440d3bf74e5b47c061727" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:19.414276 25164 raft_consensus.cc:3037] T b6ae8ea0d9e240c29296a5b826f67196 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.414638 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
W20240501 13:56:19.415568 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20240501 13:56:19.417343 25131 leader_election.cc:341] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.414276 25090 raft_consensus.cc:3037] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.422847 25166 raft_consensus.cc:2454] T e8dff22b19a440d3bf74e5b47c061727 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.424002 24983 leader_election.cc:302] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:19.429579 25209 ts_tablet_manager.cc:1398] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20240501 13:56:19.430553 25209 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.430217 25207 tablet_bootstrap.cc:654] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.424831 25269 raft_consensus.cc:504] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.404114 25450 leader_election.cc:288] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.435197 25016 raft_consensus.cc:2454] T 821bad94d33240cd9802ff777bcb842a P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:19.435488 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e059d8fd99574a5eac0b49e9405d7bd4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:19.435698 25449 raft_consensus.cc:2781] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.435977 25269 leader_election.cc:288] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
W20240501 13:56:19.436794 25131 leader_election.cc:341] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.438287 25090 raft_consensus.cc:2454] T e8dff22b19a440d3bf74e5b47c061727 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.449406 25133 leader_election.cc:302] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:19.450759 25214 raft_consensus.cc:2781] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.439159 25210 tablet_bootstrap.cc:492] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.452050 25210 ts_tablet_manager.cc:1367] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.079s user 0.018s sys 0.001s
I20240501 13:56:19.451347 25214 raft_consensus.cc:686] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:19.452998 25214 consensus_queue.cc:237] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.436367 25449 raft_consensus.cc:686] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:19.456853 25449 consensus_queue.cc:237] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.458954 25209 tablet_bootstrap.cc:654] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.436817 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e059d8fd99574a5eac0b49e9405d7bd4" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:19.457381 25210 raft_consensus.cc:348] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.461246 25210 raft_consensus.cc:374] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.461925 25210 raft_consensus.cc:729] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.462888 25210 consensus_queue.cc:260] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.461213 25018 raft_consensus.cc:3037] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.465590 25210 ts_tablet_manager.cc:1398] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 13:56:19.472921 25210 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.513265 25217 raft_consensus.cc:482] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.513746 25018 raft_consensus.cc:2454] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:19.513785 25217 raft_consensus.cc:504] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.529054 24916 catalog_manager.cc:5484] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:19.532717 25217 leader_election.cc:288] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.534415 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.535259 25016 raft_consensus.cc:3037] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:19.536876 25058 leader_election.cc:341] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.538627 25414 raft_consensus.cc:482] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.538034 24914 catalog_manager.cc:5484] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:19.539091 25414 raft_consensus.cc:504] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.540437 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.541059 25414 leader_election.cc:288] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.542973 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:19.545233 25059 leader_election.cc:341] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.545704 25059 leader_election.cc:302] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.546957 25414 raft_consensus.cc:2726] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.548908 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1954780552c46cb829f499d56d225c7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.550915 25217 raft_consensus.cc:482] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.551420 25217 raft_consensus.cc:504] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.553973 25217 leader_election.cc:288] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.554250 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:19.555286 25058 leader_election.cc:341] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.555512 25409 consensus_queue.cc:1035] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.557475 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.559064 25059 leader_election.cc:302] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.560876 25209 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.561465 25209 ts_tablet_manager.cc:1367] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.131s user 0.008s sys 0.014s
I20240501 13:56:19.562178 25133 leader_election.cc:302] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:19.568864 25210 tablet_bootstrap.cc:654] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.568514 25269 raft_consensus.cc:2781] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.570502 25269 raft_consensus.cc:686] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:19.571120 25207 tablet_bootstrap.cc:492] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.571650 25207 ts_tablet_manager.cc:1367] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.186s user 0.025s sys 0.043s
I20240501 13:56:19.571326 25269 consensus_queue.cc:237] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.571841 25217 raft_consensus.cc:2726] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.574295 25207 raft_consensus.cc:348] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.575021 25207 raft_consensus.cc:374] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.575320 25207 raft_consensus.cc:729] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.575915 25207 consensus_queue.cc:260] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:19.577311 25059 leader_election.cc:341] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.577170 25209 raft_consensus.cc:348] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.581300 25209 raft_consensus.cc:374] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.581637 25209 raft_consensus.cc:729] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.582207 25209 consensus_queue.cc:260] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.583540 25207 ts_tablet_manager.cc:1398] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 13:56:19.578042 25059 leader_election.cc:302] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.586465 25217 raft_consensus.cc:2726] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.588541 25209 ts_tablet_manager.cc:1398] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20240501 13:56:19.588938 25207 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.589375 25209 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.595031 25318 consensus_queue.cc:1035] T 17458082ba194fc8b534019c9eab0485 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.598790 25210 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.599356 25210 ts_tablet_manager.cc:1367] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.127s user 0.005s sys 0.011s
I20240501 13:56:19.602131 25210 raft_consensus.cc:348] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.602967 25210 raft_consensus.cc:374] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.603389 25210 raft_consensus.cc:729] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.607126 25166 raft_consensus.cc:3037] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.602710 25207 tablet_bootstrap.cc:654] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.609392 25409 consensus_queue.cc:1035] T 72f1ef588c2a42b4b724543c340ab182 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.604185 25210 consensus_queue.cc:260] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.620827 25207 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.621340 25207 ts_tablet_manager.cc:1367] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.001s
I20240501 13:56:19.623054 25210 ts_tablet_manager.cc:1398] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20240501 13:56:19.625392 25210 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.631903 25209 tablet_bootstrap.cc:654] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.623642 25414 consensus_queue.cc:1035] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.633636 25488 consensus_queue.cc:1035] T 4e912ea2c2074a20b15c82376861b3f7 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.634946 25269 raft_consensus.cc:482] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.624325 25207 raft_consensus.cc:348] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.635447 25269 raft_consensus.cc:504] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.637524 25269 leader_election.cc:288] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.638345 25207 raft_consensus.cc:374] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.638648 25207 raft_consensus.cc:729] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.638872 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.640198 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e912ea2c2074a20b15c82376861b3f7" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.642462 25131 leader_election.cc:302] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:19.643357 24914 catalog_manager.cc:5484] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:19.645017 25090 raft_consensus.cc:3037] T c1954780552c46cb829f499d56d225c7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.646507 25207 consensus_queue.cc:260] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.652724 25207 ts_tablet_manager.cc:1398] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20240501 13:56:19.653659 25207 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.660861 25210 tablet_bootstrap.cc:654] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.664328 25449 raft_consensus.cc:482] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.665066 25449 raft_consensus.cc:504] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.666571 25207 tablet_bootstrap.cc:654] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.667044 25449 leader_election.cc:288] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.675460 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.681550 25289 raft_consensus.cc:2726] T 4e912ea2c2074a20b15c82376861b3f7 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:56:19.689760 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17458082ba194fc8b534019c9eab0485" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.701483 24983 leader_election.cc:302] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.714685 25209 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.715401 25209 ts_tablet_manager.cc:1367] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.126s user 0.021s sys 0.017s
I20240501 13:56:19.708961 25449 raft_consensus.cc:2726] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.744258 25210 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.744963 25210 ts_tablet_manager.cc:1367] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.006s
I20240501 13:56:19.745399 25207 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.745838 25207 ts_tablet_manager.cc:1367] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.092s user 0.012s sys 0.011s
I20240501 13:56:19.748961 25209 raft_consensus.cc:348] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.749730 25209 raft_consensus.cc:374] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.749995 25209 raft_consensus.cc:729] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.750726 25209 consensus_queue.cc:260] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.753046 25209 ts_tablet_manager.cc:1398] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.037s user 0.005s sys 0.002s
I20240501 13:56:19.754293 25209 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.754412 25207 raft_consensus.cc:348] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.755842 25207 raft_consensus.cc:374] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.756127 25207 raft_consensus.cc:729] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.757014 25210 raft_consensus.cc:348] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.756939 25207 consensus_queue.cc:260] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.757694 25210 raft_consensus.cc:374] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.757948 25210 raft_consensus.cc:729] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.759568 25207 ts_tablet_manager.cc:1398] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20240501 13:56:19.760237 25209 tablet_bootstrap.cc:654] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.760490 25207 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.766381 25207 tablet_bootstrap.cc:654] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.778944 25207 tablet_bootstrap.cc:492] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:19.779414 25207 ts_tablet_manager.cc:1367] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20240501 13:56:19.782117 25207 raft_consensus.cc:348] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.782825 25207 raft_consensus.cc:374] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.783113 25207 raft_consensus.cc:729] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.783149 25210 consensus_queue.cc:260] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.784420 25207 consensus_queue.cc:260] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.786326 25210 ts_tablet_manager.cc:1398] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20240501 13:56:19.787232 25210 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.791604 25207 ts_tablet_manager.cc:1398] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240501 13:56:19.793089 25207 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:19.793238 25210 tablet_bootstrap.cc:654] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.797303 25209 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.800305 25209 ts_tablet_manager.cc:1367] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.012s
I20240501 13:56:19.803170 25409 raft_consensus.cc:482] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.803349 25209 raft_consensus.cc:348] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.803714 25409 raft_consensus.cc:504] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.804267 25209 raft_consensus.cc:374] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.804680 25209 raft_consensus.cc:729] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.805661 25409 leader_election.cc:288] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.805331 25209 consensus_queue.cc:260] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.808804 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.809509 25090 raft_consensus.cc:2454] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:19.810725 24982 leader_election.cc:302] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:19.811568 25500 raft_consensus.cc:2781] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:19.812112 25500 raft_consensus.cc:482] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:19.812470 25500 raft_consensus.cc:3037] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.814371 25209 ts_tablet_manager.cc:1398] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.014s user 0.000s sys 0.005s
I20240501 13:56:19.815227 25209 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:19.820791 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.821735 25164 raft_consensus.cc:2454] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:19.816489 25214 raft_consensus.cc:482] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.826526 25294 raft_consensus.cc:482] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.827025 25294 raft_consensus.cc:504] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.829027 25294 leader_election.cc:288] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.823302 25017 raft_consensus.cc:3037] T 17458082ba194fc8b534019c9eab0485 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.831617 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.832423 25090 raft_consensus.cc:2454] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:19.833523 25131 leader_election.cc:302] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:19.834272 25207 tablet_bootstrap.cc:654] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.834321 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.834935 25018 raft_consensus.cc:2454] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:19.837766 25294 raft_consensus.cc:2781] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:19.838373 25294 raft_consensus.cc:482] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:19.839015 25294 raft_consensus.cc:3037] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.824309 25214 raft_consensus.cc:504] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.844938 25214 leader_election.cc:288] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.847074 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.862392 25492 raft_consensus.cc:482] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.862759 25318 raft_consensus.cc:482] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.863550 25318 raft_consensus.cc:504] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.865693 25318 leader_election.cc:288] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.863309 25492 raft_consensus.cc:504] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:19.866878 25133 leader_election.cc:341] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.874308 25492 leader_election.cc:288] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.874811 25209 tablet_bootstrap.cc:654] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.876529 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.877677 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.879638 25294 raft_consensus.cc:504] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.866031 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:19.881497 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:19.881820 25131 leader_election.cc:341] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.882263 25131 leader_election.cc:302] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
W20240501 13:56:19.883492 25058 leader_election.cc:341] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.884279 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.884533 25269 raft_consensus.cc:2726] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.886013 25210 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:19.886456 25210 ts_tablet_manager.cc:1367] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.099s user 0.018s sys 0.026s
W20240501 13:56:19.884435 25059 leader_election.cc:341] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:19.887369 25059 leader_election.cc:341] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:19.888278 25058 leader_election.cc:341] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:19.890832 25058 leader_election.cc:302] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.891144 25500 raft_consensus.cc:504] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.891929 25492 raft_consensus.cc:2726] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.887774 25059 leader_election.cc:302] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.894342 25210 raft_consensus.cc:348] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.895056 25210 raft_consensus.cc:374] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.895756 25210 raft_consensus.cc:729] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.896593 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:19.898691 25210 consensus_queue.cc:260] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.900568 25214 consensus_queue.cc:1035] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:19.903033 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:19.903903 25162 raft_consensus.cc:3037] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.892745 25492 raft_consensus.cc:2726] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.895349 25500 leader_election.cc:288] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.895032 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc140167ab54483b3db40ba7a61937f" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:19.906147 25016 raft_consensus.cc:3037] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.912587 25210 ts_tablet_manager.cc:1398] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.026s user 0.006s sys 0.003s
I20240501 13:56:19.914624 25090 raft_consensus.cc:3037] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.917480 25210 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:19.919860 25414 raft_consensus.cc:482] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:19.924628 25294 leader_election.cc:288] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:19.920428 25414 raft_consensus.cc:504] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.928807 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021330a06d7d4faf9350427cbf308ed9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:19.929491 25092 raft_consensus.cc:3037] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.931432 25090 raft_consensus.cc:2454] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.932550 24982 leader_election.cc:302] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:19.933516 25014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:19.934581 25450 raft_consensus.cc:2781] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.935066 25450 raft_consensus.cc:686] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:19.936173 25450 consensus_queue.cc:237] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.937412 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "821bad94d33240cd9802ff777bcb842a" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:19.939489 25162 raft_consensus.cc:2454] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:19.946641 25016 raft_consensus.cc:2454] T 021330a06d7d4faf9350427cbf308ed9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:19.947741 25092 raft_consensus.cc:2454] T 021330a06d7d4faf9350427cbf308ed9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:19.939868 25059 leader_election.cc:302] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:19.939998 25414 leader_election.cc:288] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:19.957530 25133 leader_election.cc:302] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:19.960068 25217 raft_consensus.cc:2726] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:19.962633 25294 raft_consensus.cc:2781] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:19.964434 25294 raft_consensus.cc:686] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:19.965258 25294 consensus_queue.cc:237] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.971130 25088 raft_consensus.cc:3037] T 821bad94d33240cd9802ff777bcb842a P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:19.971488 25450 consensus_queue.cc:1035] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:19.983363 25209 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:19.983919 25209 ts_tablet_manager.cc:1367] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.169s user 0.015s sys 0.017s
I20240501 13:56:19.987486 25210 tablet_bootstrap.cc:654] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:19.988708 24916 catalog_manager.cc:5484] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:19.997614 25209 raft_consensus.cc:348] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:19.998370 25209 raft_consensus.cc:374] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:19.998864 25209 raft_consensus.cc:729] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:19.999505 25209 consensus_queue.cc:260] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.003163 25207 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.003986 25207 ts_tablet_manager.cc:1367] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.211s user 0.024s sys 0.000s
I20240501 13:56:19.998265 25269 consensus_queue.cc:1035] T 821bad94d33240cd9802ff777bcb842a P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:20.003182 25209 ts_tablet_manager.cc:1398] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240501 13:56:20.007329 25209 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.015980 25214 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:20.018658 24916 catalog_manager.cc:5484] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:20.025842 25207 raft_consensus.cc:348] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.026829 25207 raft_consensus.cc:374] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.027191 25207 raft_consensus.cc:729] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.027930 25207 consensus_queue.cc:260] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.030687 25207 ts_tablet_manager.cc:1398] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240501 13:56:20.031864 25207 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.042254 25210 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.042717 25210 ts_tablet_manager.cc:1367] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.125s user 0.010s sys 0.005s
I20240501 13:56:20.053721 25409 consensus_queue.cc:1035] T e8dff22b19a440d3bf74e5b47c061727 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.054539 25210 raft_consensus.cc:348] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.055244 25210 raft_consensus.cc:374] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.055536 25210 raft_consensus.cc:729] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.056146 25210 consensus_queue.cc:260] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.057958 25209 tablet_bootstrap.cc:654] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.065667 25210 ts_tablet_manager.cc:1398] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20240501 13:56:20.066651 25210 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.088683 25409 raft_consensus.cc:482] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.089702 25207 tablet_bootstrap.cc:654] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.089237 25409 raft_consensus.cc:504] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.095834 25409 leader_election.cc:288] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.099469 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:20.103989 24983 leader_election.cc:341] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.104949 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.105599 25090 raft_consensus.cc:2454] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
W20240501 13:56:20.108368 25131 consensus_peers.cc:485] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 -> Peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Couldn't send request to peer 171c5cc5f6df4820bc16946f8717737e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:20.109411 24982 leader_election.cc:302] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:20.110471 25409 raft_consensus.cc:2781] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.110831 25409 raft_consensus.cc:482] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.111167 25409 raft_consensus.cc:3037] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.113561 25209 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.114284 25209 ts_tablet_manager.cc:1367] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.107s user 0.014s sys 0.007s
I20240501 13:56:20.117883 25209 raft_consensus.cc:348] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.118999 25209 raft_consensus.cc:374] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.119428 25209 raft_consensus.cc:729] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.120435 25209 consensus_queue.cc:260] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.123816 25209 ts_tablet_manager.cc:1398] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:56:20.128208 25210 tablet_bootstrap.cc:654] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.132139 25409 raft_consensus.cc:504] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.135357 25207 tablet_bootstrap.cc:492] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.135831 25207 ts_tablet_manager.cc:1367] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.104s user 0.021s sys 0.015s
I20240501 13:56:20.137213 25409 leader_election.cc:288] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.138638 25207 raft_consensus.cc:348] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.140121 25207 raft_consensus.cc:374] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.140154 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.140581 25207 raft_consensus.cc:729] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.141309 25092 raft_consensus.cc:3037] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.153380 25214 raft_consensus.cc:482] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.155030 25214 raft_consensus.cc:504] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.158619 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587937b292d4407094174d4d1e073716" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.161943 25209 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.141546 25207 consensus_queue.cc:260] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:20.162279 24983 leader_election.cc:341] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.163192 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.164175 25207 ts_tablet_manager.cc:1398] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.028s user 0.004s sys 0.004s
I20240501 13:56:20.168753 25207 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
W20240501 13:56:20.167305 25131 leader_election.cc:341] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.170591 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:20.173127 25133 leader_election.cc:341] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.173583 25133 leader_election.cc:302] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.174646 25294 raft_consensus.cc:2726] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:20.175861 25214 leader_election.cc:288] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.152647 25492 raft_consensus.cc:482] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.153542 25092 raft_consensus.cc:2454] T 587937b292d4407094174d4d1e073716 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.178260 25207 tablet_bootstrap.cc:654] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.179744 24982 leader_election.cc:302] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:20.181619 25409 raft_consensus.cc:2781] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.182096 25409 raft_consensus.cc:686] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:20.177165 25492 raft_consensus.cc:504] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.182988 25409 consensus_queue.cc:237] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.184813 25492 leader_election.cc:288] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.185050 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.186867 25210 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.186925 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.187453 25210 ts_tablet_manager.cc:1367] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.013s
W20240501 13:56:20.188386 25058 leader_election.cc:341] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.189849 25210 raft_consensus.cc:348] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.190589 25210 raft_consensus.cc:374] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.190848 25210 raft_consensus.cc:729] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
W20240501 13:56:20.191692 25059 leader_election.cc:341] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.192148 25059 leader_election.cc:302] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:20.193044 25209 tablet_bootstrap.cc:654] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.197378 25210 consensus_queue.cc:260] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.198460 25414 raft_consensus.cc:2726] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:20.199491 25210 ts_tablet_manager.cc:1398] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 13:56:20.200543 25210 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.204710 25450 raft_consensus.cc:482] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.205261 25450 raft_consensus.cc:504] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.206029 24916 catalog_manager.cc:5484] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:20.207134 25450 leader_election.cc:288] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.208282 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.208737 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:20.210100 24983 leader_election.cc:341] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:20.210696 24982 leader_election.cc:341] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.211120 24982 leader_election.cc:302] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:20.212744 25409 raft_consensus.cc:2726] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:20.203325 25207 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.213260 25210 tablet_bootstrap.cc:654] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.213826 25207 ts_tablet_manager.cc:1367] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.045s user 0.016s sys 0.000s
I20240501 13:56:20.217175 25207 raft_consensus.cc:348] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.218006 25207 raft_consensus.cc:374] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.218286 25207 raft_consensus.cc:729] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.219267 25207 consensus_queue.cc:260] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.222931 25207 ts_tablet_manager.cc:1398] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240501 13:56:20.223913 25207 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.229524 25207 tablet_bootstrap.cc:654] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.235100 25210 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.235627 25210 ts_tablet_manager.cc:1367] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.035s user 0.015s sys 0.003s
I20240501 13:56:20.238011 25209 tablet_bootstrap.cc:492] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.238481 25209 ts_tablet_manager.cc:1367] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.019s
I20240501 13:56:20.238829 25409 raft_consensus.cc:482] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.239356 25409 raft_consensus.cc:504] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.241365 25409 leader_election.cc:288] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.241068 25209 raft_consensus.cc:348] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.241786 25209 raft_consensus.cc:374] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.242059 25209 raft_consensus.cc:729] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.242179 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.242882 25092 raft_consensus.cc:2454] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:20.242844 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.243206 25209 consensus_queue.cc:260] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.243640 25161 raft_consensus.cc:2454] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:20.243921 24982 leader_election.cc:302] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.244005 25210 raft_consensus.cc:348] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.245676 25409 raft_consensus.cc:2781] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.245739 25210 raft_consensus.cc:374] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.245944 25209 ts_tablet_manager.cc:1398] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240501 13:56:20.246132 25409 raft_consensus.cc:482] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.246196 25210 raft_consensus.cc:729] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.246497 25207 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.246642 25409 raft_consensus.cc:3037] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.246987 25207 ts_tablet_manager.cc:1367] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
I20240501 13:56:20.247068 25210 consensus_queue.cc:260] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.249634 25207 raft_consensus.cc:348] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.250362 25207 raft_consensus.cc:374] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.250615 25207 raft_consensus.cc:729] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.251216 25207 consensus_queue.cc:260] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.253360 25207 ts_tablet_manager.cc:1398] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 13:56:20.255465 25207 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.256593 25209 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.258491 25210 ts_tablet_manager.cc:1398] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240501 13:56:20.259469 25210 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.259305 25409 raft_consensus.cc:504] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.260664 25214 raft_consensus.cc:482] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.261123 25214 raft_consensus.cc:504] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.269170 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.269631 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.270501 25088 raft_consensus.cc:2454] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:20.269892 25018 raft_consensus.cc:2454] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:20.267889 25214 leader_election.cc:288] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.272991 25133 leader_election.cc:302] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:20.283963 25289 raft_consensus.cc:2781] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.291662 25289 raft_consensus.cc:482] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.302070 25289 raft_consensus.cc:3037] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.307538 25207 tablet_bootstrap.cc:654] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.308566 25209 tablet_bootstrap.cc:654] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.309621 25409 leader_election.cc:288] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.310945 25210 tablet_bootstrap.cc:654] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.310701 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.311378 25161 raft_consensus.cc:3037] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.330614 25161 raft_consensus.cc:2454] T 94e26289285d4f14b97aa17f829a7c96 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.331982 25209 tablet_bootstrap.cc:492] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.330734 25289 raft_consensus.cc:504] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.331619 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e26289285d4f14b97aa17f829a7c96" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.334151 25209 ts_tablet_manager.cc:1367] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.005s
I20240501 13:56:20.334286 25088 raft_consensus.cc:3037] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.336802 25209 raft_consensus.cc:348] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.337527 25209 raft_consensus.cc:374] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.337790 25209 raft_consensus.cc:729] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.338404 25209 consensus_queue.cc:260] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.340596 25209 ts_tablet_manager.cc:1398] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 13:56:20.340687 25207 tablet_bootstrap.cc:492] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.341502 25209 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.341774 25207 ts_tablet_manager.cc:1367] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.007s
I20240501 13:56:20.342747 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.343524 25092 raft_consensus.cc:3037] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.344599 25207 raft_consensus.cc:348] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.345419 25207 raft_consensus.cc:374] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.345732 25210 tablet_bootstrap.cc:492] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.345728 25207 raft_consensus.cc:729] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.346328 25210 ts_tablet_manager.cc:1367] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.087s user 0.018s sys 0.002s
I20240501 13:56:20.346745 25207 consensus_queue.cc:260] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.347741 24983 leader_election.cc:302] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:20.351559 25289 leader_election.cc:288] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.350987 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f1b26447b54d048d1ce0620e00ef85" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.352099 25450 raft_consensus.cc:2781] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.352291 25018 raft_consensus.cc:3037] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.351130 25092 raft_consensus.cc:2454] T 52f1b26447b54d048d1ce0620e00ef85 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:20.352710 25450 raft_consensus.cc:686] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:20.352680 25210 raft_consensus.cc:348] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.353463 25210 raft_consensus.cc:374] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.353560 25131 leader_election.cc:302] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:20.353760 25210 raft_consensus.cc:729] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.351037 25209 tablet_bootstrap.cc:654] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.354727 25289 raft_consensus.cc:2781] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.354305 25450 consensus_queue.cc:237] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.355494 25289 raft_consensus.cc:686] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:20.356812 25217 raft_consensus.cc:482] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.357317 25088 raft_consensus.cc:2454] T 94e26289285d4f14b97aa17f829a7c96 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.354434 25210 consensus_queue.cc:260] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.357268 25217 raft_consensus.cc:504] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.359205 25210 ts_tablet_manager.cc:1398] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240501 13:56:20.356412 25289 consensus_queue.cc:237] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.359323 25217 leader_election.cc:288] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.360110 25210 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.362752 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.363432 25017 raft_consensus.cc:2454] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:20.366721 25210 tablet_bootstrap.cc:654] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.366896 25059 leader_election.cc:302] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.367846 25217 raft_consensus.cc:2781] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.368263 25217 raft_consensus.cc:482] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.368693 25217 raft_consensus.cc:3037] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.369824 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.370616 25161 raft_consensus.cc:2454] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:20.370714 25209 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.371407 25209 ts_tablet_manager.cc:1367] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.007s
I20240501 13:56:20.374447 25209 raft_consensus.cc:348] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.375149 25209 raft_consensus.cc:374] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.375443 25209 raft_consensus.cc:729] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.376209 25207 ts_tablet_manager.cc:1398] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20240501 13:56:20.376047 25209 consensus_queue.cc:260] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.381842 25207 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.382548 25209 ts_tablet_manager.cc:1398] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240501 13:56:20.384464 25209 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.391690 25209 tablet_bootstrap.cc:654] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.392338 25217 raft_consensus.cc:504] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.400628 25217 leader_election.cc:288] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.388067 24915 catalog_manager.cc:5484] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:20.409185 25018 raft_consensus.cc:2454] T 52f1b26447b54d048d1ce0620e00ef85 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:20.404183 25207 tablet_bootstrap.cc:654] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.415874 25492 raft_consensus.cc:482] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.416412 25492 raft_consensus.cc:504] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.406133 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.418365 25492 leader_election.cc:288] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.418426 25016 raft_consensus.cc:3037] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.404245 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b63b485dd1847be84f0345d2ec1fb94" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.420377 25161 raft_consensus.cc:3037] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.407814 25210 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.421403 25210 ts_tablet_manager.cc:1367] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.004s
I20240501 13:56:20.426703 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.427423 25167 raft_consensus.cc:2454] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:20.427716 25210 raft_consensus.cc:348] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.428443 25210 raft_consensus.cc:374] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.428705 25210 raft_consensus.cc:729] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.428615 25058 leader_election.cc:302] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:20.429821 25492 raft_consensus.cc:2781] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.429509 25210 consensus_queue.cc:260] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.431269 25210 ts_tablet_manager.cc:1398] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240501 13:56:20.432214 25210 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.430223 25492 raft_consensus.cc:482] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.434361 25492 raft_consensus.cc:3037] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.424974 25450 raft_consensus.cc:482] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.441077 25450 raft_consensus.cc:504] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.443085 25450 leader_election.cc:288] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.430716 24914 catalog_manager.cc:5484] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:20.434926 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.444334 25018 raft_consensus.cc:2454] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:20.444753 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.445546 25088 raft_consensus.cc:2454] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:20.448258 24982 leader_election.cc:302] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.449126 25500 raft_consensus.cc:2781] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.449477 25500 raft_consensus.cc:482] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.449803 25500 raft_consensus.cc:3037] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.451728 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.453415 25450 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:56:20.453768 24983 leader_election.cc:341] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.454177 25161 raft_consensus.cc:2454] T 6b63b485dd1847be84f0345d2ec1fb94 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:20.455368 25492 raft_consensus.cc:504] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.457856 25058 leader_election.cc:302] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:20.459702 25414 raft_consensus.cc:2781] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.460153 25414 raft_consensus.cc:686] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:20.461002 25414 consensus_queue.cc:237] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.462266 25500 raft_consensus.cc:504] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.463027 25492 leader_election.cc:288] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.464843 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.465456 25210 tablet_bootstrap.cc:654] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.465471 25088 raft_consensus.cc:3037] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.466776 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.467360 25017 raft_consensus.cc:3037] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.470036 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f748bef4f6f64708b4a9fe990a22623f" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.470678 25162 raft_consensus.cc:3037] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.474187 25017 raft_consensus.cc:2454] T f748bef4f6f64708b4a9fe990a22623f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:20.474627 25500 leader_election.cc:288] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.475294 25214 raft_consensus.cc:482] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.475536 25088 raft_consensus.cc:2454] T e0be3b84fe26485ea767bfa7d40aa018 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.475812 25214 raft_consensus.cc:504] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.476004 25059 leader_election.cc:302] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.477149 25217 raft_consensus.cc:2781] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.478210 25209 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.477650 25217 raft_consensus.cc:686] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:20.478729 25209 ts_tablet_manager.cc:1367] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.095s user 0.019s sys 0.024s
I20240501 13:56:20.479491 24982 leader_election.cc:302] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.479216 25217 consensus_queue.cc:237] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.480422 25500 raft_consensus.cc:2781] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.483152 25214 leader_election.cc:288] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.484156 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.485610 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.487172 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:20.488067 24983 leader_election.cc:341] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.488704 25409 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:56:20.488173 25133 leader_election.cc:341] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:20.494251 25131 leader_election.cc:341] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.494537 25016 raft_consensus.cc:2454] T 6b63b485dd1847be84f0345d2ec1fb94 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:20.495656 25131 leader_election.cc:302] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.496694 25214 raft_consensus.cc:2726] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:20.497151 25162 raft_consensus.cc:2454] T f748bef4f6f64708b4a9fe990a22623f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:20.497592 25500 raft_consensus.cc:686] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:20.498513 25500 consensus_queue.cc:237] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.506032 25214 consensus_queue.cc:1035] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.513432 25209 raft_consensus.cc:348] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.514206 25209 raft_consensus.cc:374] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.514503 25209 raft_consensus.cc:729] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.515094 25209 consensus_queue.cc:260] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.525677 25209 ts_tablet_manager.cc:1398] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20240501 13:56:20.526577 25209 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.557050 25289 consensus_queue.cc:1035] T 021330a06d7d4faf9350427cbf308ed9 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:20.562557 25209 tablet_bootstrap.cc:654] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.573415 24915 catalog_manager.cc:5484] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
W20240501 13:56:20.577723 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:20.579035 25207 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.579537 25207 ts_tablet_manager.cc:1367] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.198s user 0.024s sys 0.020s
I20240501 13:56:20.582312 25207 raft_consensus.cc:348] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.582980 25207 raft_consensus.cc:374] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.583254 25207 raft_consensus.cc:729] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.583918 25207 consensus_queue.cc:260] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.586970 25207 ts_tablet_manager.cc:1398] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:56:20.588032 25207 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.589743 25209 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.590276 25209 ts_tablet_manager.cc:1367] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.002s
I20240501 13:56:20.592693 25209 raft_consensus.cc:348] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.593510 25209 raft_consensus.cc:374] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.593819 25209 raft_consensus.cc:729] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.594638 25209 consensus_queue.cc:260] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.597471 25209 ts_tablet_manager.cc:1398] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240501 13:56:20.598053 25210 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.598459 25209 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.598567 25210 ts_tablet_manager.cc:1367] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.167s user 0.021s sys 0.021s
I20240501 13:56:20.602895 25210 raft_consensus.cc:348] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.603637 25210 raft_consensus.cc:374] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.603921 25210 raft_consensus.cc:729] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.606465 25209 tablet_bootstrap.cc:654] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.608521 25210 consensus_queue.cc:260] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.610743 25210 ts_tablet_manager.cc:1398] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 13:56:20.611631 25210 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.614423 25207 tablet_bootstrap.cc:654] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.624289 25210 tablet_bootstrap.cc:654] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.629665 25500 consensus_queue.cc:1035] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:20.641500 25207 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.642000 25207 ts_tablet_manager.cc:1367] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.054s user 0.024s sys 0.007s
I20240501 13:56:20.642254 25210 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.642705 25210 ts_tablet_manager.cc:1367] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.006s
I20240501 13:56:20.643009 25209 tablet_bootstrap.cc:492] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.643537 25209 ts_tablet_manager.cc:1367] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.009s
I20240501 13:56:20.644742 25207 raft_consensus.cc:348] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.645079 25210 raft_consensus.cc:348] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.645556 25207 raft_consensus.cc:374] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.645843 25210 raft_consensus.cc:374] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.645954 25207 raft_consensus.cc:729] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.646214 25210 raft_consensus.cc:729] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.646668 25209 raft_consensus.cc:348] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.646862 25210 consensus_queue.cc:260] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.647445 25209 raft_consensus.cc:374] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.646633 25207 consensus_queue.cc:260] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.649261 25210 ts_tablet_manager.cc:1398] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240501 13:56:20.649387 25207 ts_tablet_manager.cc:1398] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:56:20.650260 25210 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.647835 25209 raft_consensus.cc:729] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.654639 25207 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.656100 25449 raft_consensus.cc:482] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.656663 25449 raft_consensus.cc:504] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.657068 25209 consensus_queue.cc:260] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.658571 25449 leader_election.cc:288] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.659219 25209 ts_tablet_manager.cc:1398] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240501 13:56:20.660113 25209 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.660230 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.663255 25207 tablet_bootstrap.cc:654] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.663794 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.664472 25162 raft_consensus.cc:2454] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:20.663151 25092 raft_consensus.cc:2454] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:20.665047 25210 tablet_bootstrap.cc:654] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.666004 24982 leader_election.cc:302] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.666855 25449 raft_consensus.cc:2781] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.667215 25449 raft_consensus.cc:482] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.667562 25449 raft_consensus.cc:3037] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.672451 25209 tablet_bootstrap.cc:654] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.676545 25500 raft_consensus.cc:482] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.677340 25500 raft_consensus.cc:504] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.679625 24916 catalog_manager.cc:5484] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:20.681996 24916 catalog_manager.cc:5484] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:20.693114 25414 raft_consensus.cc:482] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.693835 25414 raft_consensus.cc:504] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.697569 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:20.699965 25059 leader_election.cc:341] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.701882 25414 leader_election.cc:288] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.703434 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.708020 25092 raft_consensus.cc:3037] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:20.712306 24982 leader_election.cc:341] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.714360 25500 leader_election.cc:288] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.714495 25210 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.715204 25210 ts_tablet_manager.cc:1367] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.004s
I20240501 13:56:20.719136 25162 raft_consensus.cc:3037] T 587937b292d4407094174d4d1e073716 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.724722 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.725592 25161 raft_consensus.cc:2454] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:20.726348 25210 raft_consensus.cc:348] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.727308 25210 raft_consensus.cc:374] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.727655 25210 raft_consensus.cc:729] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.728650 25058 leader_election.cc:302] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.728514 25210 consensus_queue.cc:260] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.731846 25414 raft_consensus.cc:2781] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.732228 25414 raft_consensus.cc:482] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.732590 25414 raft_consensus.cc:3037] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.736747 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:20.739028 25414 raft_consensus.cc:504] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.741226 25414 leader_election.cc:288] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.741838 25207 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.742352 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.742376 25207 ts_tablet_manager.cc:1367] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.088s user 0.020s sys 0.054s
I20240501 13:56:20.743225 25449 raft_consensus.cc:504] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:20.743671 25059 leader_election.cc:341] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.745291 25449 leader_election.cc:288] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:20.739696 25209 tablet_bootstrap.cc:492] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.745546 25207 raft_consensus.cc:348] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.746297 25209 ts_tablet_manager.cc:1367] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.086s user 0.024s sys 0.012s
I20240501 13:56:20.746448 25207 raft_consensus.cc:374] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.746824 25207 raft_consensus.cc:729] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.747057 25210 ts_tablet_manager.cc:1398] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20240501 13:56:20.748070 25210 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.747941 25207 consensus_queue.cc:260] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:20.749938 24983 leader_election.cc:341] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.750420 24983 leader_election.cc:302] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:20.753679 25209 raft_consensus.cc:348] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.757763 25207 ts_tablet_manager.cc:1398] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240501 13:56:20.757844 25209 raft_consensus.cc:374] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.758793 25207 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.753423 25500 raft_consensus.cc:2726] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:20.758288 25209 raft_consensus.cc:729] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.755713 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.764107 25088 raft_consensus.cc:3037] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.764400 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.765018 25166 raft_consensus.cc:3037] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.753630 25450 consensus_queue.cc:1035] T 587937b292d4407094174d4d1e073716 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240501 13:56:20.769245 25294 consensus_queue.cc:1035] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.770223 25088 raft_consensus.cc:2454] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.758044 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1c01c0dc4a345bbb6ce50e6f4ea3ea4" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:20.772931 25162 raft_consensus.cc:3037] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.772189 24982 leader_election.cc:302] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:20.782178 25209 consensus_queue.cc:260] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.785681 25500 raft_consensus.cc:2781] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.786201 25500 raft_consensus.cc:686] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:20.787071 25500 consensus_queue.cc:237] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.788228 25209 ts_tablet_manager.cc:1398] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.042s user 0.009s sys 0.000s
I20240501 13:56:20.789155 25209 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:20.795665 25162 raft_consensus.cc:2454] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:20.797576 25294 raft_consensus.cc:482] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.786311 25210 tablet_bootstrap.cc:654] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.798144 25294 raft_consensus.cc:504] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.798959 25207 tablet_bootstrap.cc:654] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.801748 25166 raft_consensus.cc:2454] T f348136225c444e1a2ed5658a3306eb7 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:20.802175 25294 leader_election.cc:288] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.804478 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:20.804701 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.807211 25091 raft_consensus.cc:2454] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
W20240501 13:56:20.809367 25133 leader_election.cc:341] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.811008 25131 leader_election.cc:302] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.813797 25269 raft_consensus.cc:2781] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.814222 25269 raft_consensus.cc:482] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.814538 25269 raft_consensus.cc:3037] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.811798 25294 raft_consensus.cc:482] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.815640 25294 raft_consensus.cc:504] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.811575 25058 leader_election.cc:302] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.830168 25209 tablet_bootstrap.cc:654] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.831743 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab74f4d1c64cafbbc48dc4accf5917" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:20.832749 25133 leader_election.cc:341] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.832921 25269 raft_consensus.cc:504] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.834244 25269 leader_election.cc:288] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.835211 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.835469 24914 catalog_manager.cc:5484] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
W20240501 13:56:20.836314 25133 leader_election.cc:341] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:20.830116 25294 leader_election.cc:288] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.838603 25537 raft_consensus.cc:2781] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.841849 25537 raft_consensus.cc:686] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:20.842705 25537 consensus_queue.cc:237] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.848910 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:20.849843 25093 raft_consensus.cc:3037] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.848896 25091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab74f4d1c64cafbbc48dc4accf5917" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:20.851811 25091 raft_consensus.cc:2454] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:20.853775 25131 leader_election.cc:302] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.864637 25294 raft_consensus.cc:2781] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:20.865324 25294 raft_consensus.cc:482] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:20.865916 25294 raft_consensus.cc:3037] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.867487 25093 raft_consensus.cc:2454] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:20.869788 25131 leader_election.cc:302] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:20.873932 25214 raft_consensus.cc:2781] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:20.874601 25214 raft_consensus.cc:686] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:20.875559 25214 consensus_queue.cc:237] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.873914 25209 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:20.880482 25209 ts_tablet_manager.cc:1367] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.092s user 0.014s sys 0.001s
I20240501 13:56:20.883498 25294 raft_consensus.cc:504] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.886413 25533 consensus_queue.cc:1035] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.888875 25210 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:20.890419 25210 ts_tablet_manager.cc:1367] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.006s
I20240501 13:56:20.883953 25209 raft_consensus.cc:348] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.892624 25209 raft_consensus.cc:374] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.894343 25414 consensus_queue.cc:1035] T 6b63b485dd1847be84f0345d2ec1fb94 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.895931 25209 raft_consensus.cc:729] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.894673 25450 consensus_queue.cc:1035] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.898471 25209 consensus_queue.cc:260] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.895169 25210 raft_consensus.cc:348] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.903604 25210 raft_consensus.cc:374] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.904969 25210 raft_consensus.cc:729] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.908257 25207 tablet_bootstrap.cc:492] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:20.910072 25207 ts_tablet_manager.cc:1367] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.151s user 0.014s sys 0.011s
I20240501 13:56:20.906977 25210 consensus_queue.cc:260] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.913661 25210 ts_tablet_manager.cc:1398] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240501 13:56:20.914320 25294 leader_election.cc:288] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.918882 25210 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:20.918182 25207 raft_consensus.cc:348] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.921233 25207 raft_consensus.cc:374] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:20.921563 25207 raft_consensus.cc:729] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:20.919267 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab74f4d1c64cafbbc48dc4accf5917" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:20.922287 25207 consensus_queue.cc:260] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.923579 25017 raft_consensus.cc:3037] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.926364 25207 ts_tablet_manager.cc:1398] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240501 13:56:20.930274 25207 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:20.935999 25210 tablet_bootstrap.cc:654] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.944264 25166 raft_consensus.cc:3037] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:20.945531 25207 tablet_bootstrap.cc:654] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:20.948181 25294 consensus_queue.cc:1035] T 52f1b26447b54d048d1ce0620e00ef85 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.956437 25500 consensus_queue.cc:1035] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:20.968935 24916 catalog_manager.cc:5484] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:20.979557 25500 consensus_queue.cc:1035] T e0be3b84fe26485ea767bfa7d40aa018 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:20.981273 25294 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.981822 25294 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.983794 25294 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:20.986529 25294 raft_consensus.cc:482] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:20.986961 25294 raft_consensus.cc:504] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:20.990556 25294 leader_election.cc:288] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
W20240501 13:56:20.995224 25131 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 99043.2us
I20240501 13:56:21.005259 25207 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:21.005070 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:21.005945 25207 ts_tablet_manager.cc:1367] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.010s
I20240501 13:56:21.015060 25207 raft_consensus.cc:348] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.016037 25207 raft_consensus.cc:374] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:21.007297 25017 raft_consensus.cc:2454] T 5bab74f4d1c64cafbbc48dc4accf5917 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:21.018982 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:21.016412 25207 raft_consensus.cc:729] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:21.026062 25207 consensus_queue.cc:260] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.031045 25409 consensus_queue.cc:1035] T 94e26289285d4f14b97aa17f829a7c96 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:56:21.031311 25133 leader_election.cc:341] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.033267 25133 leader_election.cc:302] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:21.035341 25214 raft_consensus.cc:2781] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:21.035979 25214 raft_consensus.cc:686] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:21.043255 25207 ts_tablet_manager.cc:1398] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.037s user 0.007s sys 0.003s
I20240501 13:56:21.047622 25207 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:21.056001 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bab74f4d1c64cafbbc48dc4accf5917" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:21.056994 25093 raft_consensus.cc:3037] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:21.036734 25214 consensus_queue.cc:237] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.096735 25210 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:21.097280 25210 ts_tablet_manager.cc:1367] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.179s user 0.014s sys 0.025s
I20240501 13:56:21.099879 25500 raft_consensus.cc:482] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.100432 25500 raft_consensus.cc:504] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.101797 25210 raft_consensus.cc:348] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.102522 25210 raft_consensus.cc:374] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:21.102787 25210 raft_consensus.cc:729] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:21.103374 25500 leader_election.cc:288] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.103497 25210 consensus_queue.cc:260] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.106458 25207 tablet_bootstrap.cc:654] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:21.111304 25093 raft_consensus.cc:2454] T 5bab74f4d1c64cafbbc48dc4accf5917 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:21.135723 25209 ts_tablet_manager.cc:1398] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.255s user 0.048s sys 0.048s
I20240501 13:56:21.137069 25209 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:21.132515 25500 raft_consensus.cc:482] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.172857 25500 raft_consensus.cc:504] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.187005 25500 leader_election.cc:288] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.175150 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:21.203857 24983 leader_election.cc:341] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.192499 25449 raft_consensus.cc:482] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.236471 25209 tablet_bootstrap.cc:654] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:21.238765 24915 catalog_manager.cc:5484] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:21.231935 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:21.249964 24983 leader_election.cc:341] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.237226 25449 raft_consensus.cc:504] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.264057 25210 ts_tablet_manager.cc:1398] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.166s user 0.013s sys 0.002s
I20240501 13:56:21.265220 25210 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:21.278414 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.269217 25575 raft_consensus.cc:482] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.308893 25583 consensus_queue.cc:1035] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:21.314345 25575 raft_consensus.cc:504] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.316709 25210 tablet_bootstrap.cc:654] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:21.318610 25574 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.319113 25574 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.323782 25574 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.324602 25207 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:21.325085 25207 ts_tablet_manager.cc:1367] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.281s user 0.037s sys 0.013s
I20240501 13:56:21.328655 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:21.329370 25017 raft_consensus.cc:2454] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:21.328789 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0be3b84fe26485ea767bfa7d40aa018" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.331810 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.333938 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.334659 25092 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:21.335328 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:21.337457 25449 leader_election.cc:288] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.340066 25059 leader_election.cc:302] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
W20240501 13:56:21.340978 25059 leader_election.cc:341] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.341742 25574 consensus_queue.cc:1035] T f748bef4f6f64708b4a9fe990a22623f P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:21.343513 25207 raft_consensus.cc:348] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.344274 25207 raft_consensus.cc:374] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:21.347565 25207 raft_consensus.cc:729] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:21.348683 25585 consensus_queue.cc:1035] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 13:56:21.345696 25575 leader_election.cc:288] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.345238 25533 raft_consensus.cc:2781] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:21.350354 25533 raft_consensus.cc:482] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:21.350724 25533 raft_consensus.cc:3037] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:21.348240 25207 consensus_queue.cc:260] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:21.352622 25133 consensus_peers.cc:485] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 -> Peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Couldn't send request to peer 346b2bb141744d459f1fa3ad602d1a15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:21.374011 25209 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:21.374533 25209 ts_tablet_manager.cc:1367] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.238s user 0.013s sys 0.003s
I20240501 13:56:21.408668 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:21.409479 25159 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:21.410799 25207 ts_tablet_manager.cc:1398] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.085s user 0.006s sys 0.001s
I20240501 13:56:21.478544 25533 raft_consensus.cc:504] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:21.484225 25058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.131:40317, user_credentials={real_user=slave}} blocked reactor thread for 143100us
I20240501 13:56:21.439806 24915 catalog_manager.cc:5484] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
W20240501 13:56:21.494210 24982 leader_election.cc:341] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.494534 25131 leader_election.cc:302] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:21.495054 25163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:21.497622 25131 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:21.499265 25563 raft_consensus.cc:2726] T e0be3b84fe26485ea767bfa7d40aa018 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:56:21.503621 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
W20240501 13:56:21.506974 24983 leader_election.cc:341] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.511020 25561 raft_consensus.cc:2781] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:21.511628 25561 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:21.512068 25561 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:21.514343 25017 raft_consensus.cc:3037] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:21.516103 25209 raft_consensus.cc:348] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.516808 25209 raft_consensus.cc:374] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:21.502344 24982 leader_election.cc:302] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:21.520191 25450 raft_consensus.cc:2726] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:21.517076 25209 raft_consensus.cc:729] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:21.523633 25209 consensus_queue.cc:260] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.530870 25561 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.533165 25561 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:21.536105 25207 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
W20240501 13:56:21.531502 25059 consensus_peers.cc:485] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e -> Peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Couldn't send request to peer 346b2bb141744d459f1fa3ad602d1a15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:21.499718 25533 leader_election.cc:288] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.547879 25209 ts_tablet_manager.cc:1398] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.173s user 0.002s sys 0.015s
I20240501 13:56:21.544006 25010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
W20240501 13:56:21.525409 24982 leader_election.cc:341] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.553148 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.555181 25088 raft_consensus.cc:2379] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:21.560624 25214 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:21.552786 25058 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:21.565171 25584 raft_consensus.cc:2781] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:21.566216 25209 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:21.567638 25584 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:21.568058 25584 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:21.571588 25133 leader_election.cc:341] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.635337 25500 raft_consensus.cc:482] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.635860 25500 raft_consensus.cc:504] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.639621 25565 raft_consensus.cc:482] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.640244 25565 raft_consensus.cc:504] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.643424 25500 leader_election.cc:288] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.643669 25210 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:21.644261 25210 ts_tablet_manager.cc:1367] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.379s user 0.013s sys 0.009s
I20240501 13:56:21.564622 24982 leader_election.cc:302] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:21.656486 25017 raft_consensus.cc:2454] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:21.660439 25059 leader_election.cc:302] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:21.661057 25085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:21.665922 25207 tablet_bootstrap.cc:654] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:21.667390 25217 raft_consensus.cc:2781] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:21.668016 25217 raft_consensus.cc:686] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:21.669922 25217 consensus_queue.cc:237] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:21.682571 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 157216us
I20240501 13:56:21.683530 25210 raft_consensus.cc:348] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.686003 25210 raft_consensus.cc:374] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:21.686404 25210 raft_consensus.cc:729] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:21.687361 25210 consensus_queue.cc:260] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.684512 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:21.690407 25564 consensus_queue.cc:1035] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 13:56:21.691632 25210 ts_tablet_manager.cc:1398] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.047s user 0.005s sys 0.002s
I20240501 13:56:21.691521 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:21.692618 25210 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:21.692469 25449 consensus_queue.cc:1035] T f1c01c0dc4a345bbb6ce50e6f4ea3ea4 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:56:21.695044 24983 leader_election.cc:341] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.685117 25450 raft_consensus.cc:2726] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:21.706036 24982 leader_election.cc:302] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:21.710930 25500 raft_consensus.cc:2726] T 4c016fa6d9a34be48e77cc49c8f93c44 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:21.714252 25017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:21.716269 25017 raft_consensus.cc:2454] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
W20240501 13:56:21.720057 25058 leader_election.cc:341] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.698377 25584 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.727694 25584 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.728839 25565 leader_election.cc:288] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:21.733482 25209 tablet_bootstrap.cc:654] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
W20240501 13:56:21.780421 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 68217.1us
W20240501 13:56:21.782605 25058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.131:40317, user_credentials={real_user=slave}} blocked reactor thread for 62420.5us
I20240501 13:56:21.788677 25165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c016fa6d9a34be48e77cc49c8f93c44" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:21.802110 25085 raft_consensus.cc:2379] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ef020bad8d24499281f69de00e90af06 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:21.861337 25583 raft_consensus.cc:482] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:21.865583 25583 raft_consensus.cc:504] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:21.869279 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:21.872340 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:21.869769 25210 tablet_bootstrap.cc:654] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:21.869424 25584 consensus_queue.cc:1035] T f348136225c444e1a2ed5658a3306eb7 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:21.943210 25583 leader_election.cc:288] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:21.945981 25207 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:21.873606 25133 leader_election.cc:302] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:21.956194 25164 raft_consensus.cc:2379] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:21.953457 25131 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:21.971791 25207 ts_tablet_manager.cc:1367] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.436s user 0.029s sys 0.035s
I20240501 13:56:21.976956 25269 raft_consensus.cc:2781] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:21.977392 25269 raft_consensus.cc:482] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:21.977763 25269 raft_consensus.cc:3037] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:21.977782 25561 raft_consensus.cc:2726] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:56:21.996721 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:21.997233 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
W20240501 13:56:21.957763 25058 leader_election.cc:341] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:21.954113 25093 raft_consensus.cc:2454] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:22.025570 25207 raft_consensus.cc:348] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.027420 25207 raft_consensus.cc:374] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.028653 25207 raft_consensus.cc:729] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
W20240501 13:56:22.034826 25058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.131:40317, user_credentials={real_user=slave}} blocked reactor thread for 77131.7us
I20240501 13:56:22.032982 25269 raft_consensus.cc:504] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.034765 25207 consensus_queue.cc:260] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.032065 25012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.047449 25207 ts_tablet_manager.cc:1398] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.074s user 0.008s sys 0.000s
I20240501 13:56:22.038864 25012 raft_consensus.cc:2454] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:22.053114 25209 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.054541 25269 leader_election.cc:288] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
W20240501 13:56:22.062400 25131 leader_election.cc:341] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
W20240501 13:56:22.064028 25059 leader_election.cc:341] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.065079 24982 leader_election.cc:302] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.067282 25210 tablet_bootstrap.cc:492] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:22.067734 25409 raft_consensus.cc:2781] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.072614 25409 raft_consensus.cc:482] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.072880 25409 raft_consensus.cc:3037] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.075876 25090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:22.080997 25210 ts_tablet_manager.cc:1367] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.389s user 0.028s sys 0.008s
I20240501 13:56:22.084038 25058 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.086632 25059 leader_election.cc:302] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:22.096155 25209 ts_tablet_manager.cc:1367] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.531s user 0.018s sys 0.022s
I20240501 13:56:22.085336 25207 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:22.112044 25409 raft_consensus.cc:504] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.101238 25210 raft_consensus.cc:348] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.113245 25210 raft_consensus.cc:374] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.113780 25210 raft_consensus.cc:729] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.103767 25209 raft_consensus.cc:348] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.114856 25209 raft_consensus.cc:374] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.105263 25584 raft_consensus.cc:2726] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.114238 25409 leader_election.cc:288] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:22.103947 25131 leader_election.cc:341] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.115149 25209 raft_consensus.cc:729] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
W20240501 13:56:22.115319 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 48026.5us
I20240501 13:56:22.118645 25209 consensus_queue.cc:260] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.118645 25210 consensus_queue.cc:260] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.129344 25533 raft_consensus.cc:2781] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.129796 25533 raft_consensus.cc:482] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.130121 25533 raft_consensus.cc:3037] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.141845 25450 raft_consensus.cc:482] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.142465 25450 raft_consensus.cc:504] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.144850 25209 ts_tablet_manager.cc:1398] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20240501 13:56:22.147037 25450 leader_election.cc:288] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.148116 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.148941 25164 raft_consensus.cc:2454] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:22.152640 25209 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.157070 25210 ts_tablet_manager.cc:1398] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.075s user 0.005s sys 0.003s
I20240501 13:56:22.158445 25210 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
W20240501 13:56:22.163722 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:22.166821 25533 raft_consensus.cc:504] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.176541 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:22.176759 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:22.177664 25092 raft_consensus.cc:2379] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:22.176640 25574 raft_consensus.cc:482] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.186177 25574 raft_consensus.cc:504] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.188284 25574 leader_election.cc:288] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
W20240501 13:56:22.189097 25133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.23.70.129:33309, user_credentials={real_user=slave}} blocked reactor thread for 340592us
I20240501 13:56:22.189299 25012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.191040 25214 consensus_queue.cc:1035] T 5bab74f4d1c64cafbbc48dc4accf5917 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 13:56:22.191964 25059 leader_election.cc:341] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.197379 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.198122 25013 raft_consensus.cc:3037] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.205096 25537 consensus_queue.cc:1035] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:22.222429 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.223927 25011 raft_consensus.cc:2379] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate 346b2bb141744d459f1fa3ad602d1a15 in this term.
I20240501 13:56:22.221099 25533 leader_election.cc:288] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.226614 25209 tablet_bootstrap.cc:654] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.227658 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.246905 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:22.247603 25162 raft_consensus.cc:3037] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.280992 25210 tablet_bootstrap.cc:654] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.286386 25013 raft_consensus.cc:2454] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
W20240501 13:56:22.287829 25058 consensus_peers.cc:485] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e -> Peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Couldn't send request to peer ef020bad8d24499281f69de00e90af06. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240501 13:56:22.291800 25058 leader_election.cc:341] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.292269 25058 leader_election.cc:302] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.295878 25574 raft_consensus.cc:2726] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.295882 25162 raft_consensus.cc:2454] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:22.301272 25058 leader_election.cc:302] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters: 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:22.302434 25414 raft_consensus.cc:2781] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.303678 25414 raft_consensus.cc:686] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:22.302930 24916 catalog_manager.cc:5484] T 4c016fa6d9a34be48e77cc49c8f93c44 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:22.304898 25414 consensus_queue.cc:237] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.307423 25207 tablet_bootstrap.cc:654] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.328511 25209 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.329021 25209 ts_tablet_manager.cc:1367] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.177s user 0.015s sys 0.009s
I20240501 13:56:22.331851 25209 raft_consensus.cc:348] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.332599 25209 raft_consensus.cc:374] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.332896 25209 raft_consensus.cc:729] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.334409 25209 consensus_queue.cc:260] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.349117 25209 ts_tablet_manager.cc:1398] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20240501 13:56:22.350309 25209 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.350046 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf58f127779f4ebd94273cd7fda138cd" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:22.351007 25161 raft_consensus.cc:2379] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:22.352546 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b159610f514496a8a1faf7d78c6c839" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.367830 25209 tablet_bootstrap.cc:654] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.372797 25011 raft_consensus.cc:3037] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.374773 25207 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:22.375166 25207 ts_tablet_manager.cc:1367] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.290s user 0.016s sys 0.006s
I20240501 13:56:22.382472 25207 raft_consensus.cc:348] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.383252 25207 raft_consensus.cc:374] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.383553 25207 raft_consensus.cc:729] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.387238 25210 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:22.387734 25210 ts_tablet_manager.cc:1367] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.230s user 0.020s sys 0.009s
I20240501 13:56:22.389127 25207 consensus_queue.cc:260] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.394407 25210 raft_consensus.cc:348] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.395431 25210 raft_consensus.cc:374] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.395802 25210 raft_consensus.cc:729] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.399469 25210 consensus_queue.cc:260] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.402338 25133 leader_election.cc:302] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e
I20240501 13:56:22.403280 25565 raft_consensus.cc:2781] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.403787 25565 raft_consensus.cc:686] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:22.405135 25210 ts_tablet_manager.cc:1398] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20240501 13:56:22.404692 25565 consensus_queue.cc:237] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.412314 24983 leader_election.cc:302] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.413633 25594 raft_consensus.cc:2726] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.413849 24983 leader_election.cc:302] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.414710 25594 raft_consensus.cc:2726] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.416306 25207 ts_tablet_manager.cc:1398] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20240501 13:56:22.417740 25210 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:22.418486 25209 tablet_bootstrap.cc:492] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.418892 25207 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:22.421983 25209 ts_tablet_manager.cc:1367] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.009s
I20240501 13:56:22.427549 25210 tablet_bootstrap.cc:654] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.428671 25209 raft_consensus.cc:348] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.437513 24916 catalog_manager.cc:5484] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:22.452632 25209 raft_consensus.cc:374] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.452975 25209 raft_consensus.cc:729] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.453625 25209 consensus_queue.cc:260] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.455885 25209 ts_tablet_manager.cc:1398] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.034s user 0.002s sys 0.002s
I20240501 13:56:22.456925 25209 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.470204 25210 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:22.470762 25210 ts_tablet_manager.cc:1367] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.009s
I20240501 13:56:22.439903 25207 tablet_bootstrap.cc:654] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.441346 24914 catalog_manager.cc:5484] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:22.473455 25210 raft_consensus.cc:348] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.475735 25210 raft_consensus.cc:374] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.476002 25210 raft_consensus.cc:729] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.480958 25209 tablet_bootstrap.cc:654] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.482604 25210 consensus_queue.cc:260] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.499284 25537 raft_consensus.cc:482] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.499823 25537 raft_consensus.cc:504] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.509258 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:22.510547 25058 leader_election.cc:341] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.511073 25537 leader_election.cc:288] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.514828 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
W20240501 13:56:22.516083 25059 leader_election.cc:341] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.516554 25059 leader_election.cc:302] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.517526 25537 raft_consensus.cc:2726] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.518325 25207 tablet_bootstrap.cc:492] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:22.520049 25210 ts_tablet_manager.cc:1398] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20240501 13:56:22.520140 25207 ts_tablet_manager.cc:1367] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.006s
I20240501 13:56:22.521142 25210 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:22.534092 25209 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.535199 25209 ts_tablet_manager.cc:1367] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.004s
I20240501 13:56:22.534660 25210 tablet_bootstrap.cc:654] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.536088 25207 raft_consensus.cc:348] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.536731 25207 raft_consensus.cc:374] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.536945 25207 raft_consensus.cc:729] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.537571 25207 consensus_queue.cc:260] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.539395 25162 raft_consensus.cc:3037] T 4c016fa6d9a34be48e77cc49c8f93c44 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.540969 25209 raft_consensus.cc:348] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.541810 25209 raft_consensus.cc:374] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.542079 25209 raft_consensus.cc:729] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.543073 25209 consensus_queue.cc:260] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.545493 25209 ts_tablet_manager.cc:1398] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 13:56:22.549080 25209 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.550527 25207 ts_tablet_manager.cc:1398] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240501 13:56:22.551592 25207 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:22.554770 25209 tablet_bootstrap.cc:654] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.567349 25207 tablet_bootstrap.cc:654] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.571991 25210 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:22.572558 25210 ts_tablet_manager.cc:1367] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.010s
I20240501 13:56:22.575165 25210 raft_consensus.cc:348] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.575904 25210 raft_consensus.cc:374] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.576205 25210 raft_consensus.cc:729] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.576918 25210 consensus_queue.cc:260] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.583392 25209 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.583897 25209 ts_tablet_manager.cc:1367] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.000s
I20240501 13:56:22.586410 25209 raft_consensus.cc:348] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.587126 25209 raft_consensus.cc:374] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.587421 25209 raft_consensus.cc:729] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.588114 25207 tablet_bootstrap.cc:492] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:22.588019 25209 consensus_queue.cc:260] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.588599 25210 ts_tablet_manager.cc:1398] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20240501 13:56:22.588614 25207 ts_tablet_manager.cc:1367] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.005s
I20240501 13:56:22.589644 25210 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e: Bootstrap starting.
I20240501 13:56:22.590262 25209 ts_tablet_manager.cc:1398] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 13:56:22.591660 25207 raft_consensus.cc:348] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.592406 25207 raft_consensus.cc:374] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.592293 25414 raft_consensus.cc:482] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.592888 25414 raft_consensus.cc:504] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.594811 25414 leader_election.cc:288] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.596004 25210 tablet_bootstrap.cc:654] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.596143 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.596886 25011 raft_consensus.cc:2454] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:22.597271 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.597887 25159 raft_consensus.cc:2454] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:22.598023 25059 leader_election.cc:302] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:22.598796 25414 raft_consensus.cc:2781] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.599143 25414 raft_consensus.cc:482] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.599480 25414 raft_consensus.cc:3037] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.600654 25574 raft_consensus.cc:482] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.601148 25574 raft_consensus.cc:504] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.603180 25574 leader_election.cc:288] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.605329 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.606607 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e1a6d9d3d9424eb9fb5a781214940e" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.607477 25011 raft_consensus.cc:2379] T b6e1a6d9d3d9424eb9fb5a781214940e P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171c5cc5f6df4820bc16946f8717737e in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:22.608974 25059 leader_election.cc:302] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e; no voters: 346b2bb141744d459f1fa3ad602d1a15, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.612449 25537 raft_consensus.cc:3037] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.620297 25209 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.626497 25209 tablet_bootstrap.cc:654] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.627828 25207 raft_consensus.cc:729] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.628614 25207 consensus_queue.cc:260] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.637764 25537 raft_consensus.cc:2726] T b6e1a6d9d3d9424eb9fb5a781214940e P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.631682 25414 raft_consensus.cc:504] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.638940 25207 ts_tablet_manager.cc:1398] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20240501 13:56:22.644944 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.645555 25011 raft_consensus.cc:3037] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.647361 25207 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:22.648952 25414 leader_election.cc:288] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.652631 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:22.653432 25159 raft_consensus.cc:3037] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.672708 25594 raft_consensus.cc:482] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.673281 25594 raft_consensus.cc:504] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.675621 25594 leader_election.cc:288] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.676224 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b085a215235d482e86b0a9529458b3c6" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:22.676959 25093 raft_consensus.cc:2454] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 0.
I20240501 13:56:22.677973 24982 leader_election.cc:302] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:22.678205 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b085a215235d482e86b0a9529458b3c6" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.678745 25409 raft_consensus.cc:2781] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.679108 25409 raft_consensus.cc:482] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.679428 25409 raft_consensus.cc:3037] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
W20240501 13:56:22.681885 24983 leader_election.cc:341] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.683470 25159 raft_consensus.cc:2454] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:22.684510 25058 leader_election.cc:302] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:22.684955 25207 tablet_bootstrap.cc:654] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.685186 25537 raft_consensus.cc:2781] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.685603 25537 raft_consensus.cc:686] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:22.685983 25409 raft_consensus.cc:504] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.686523 25537 consensus_queue.cc:237] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.689381 25210 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e: No bootstrap required, opened a new log
I20240501 13:56:22.689891 25210 ts_tablet_manager.cc:1367] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e: Time spent bootstrapping tablet: real 0.100s user 0.018s sys 0.007s
I20240501 13:56:22.693205 25210 raft_consensus.cc:348] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.693984 25210 raft_consensus.cc:374] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.694279 25210 raft_consensus.cc:729] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.678771 25011 raft_consensus.cc:2454] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:22.698019 25209 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.698558 25209 ts_tablet_manager.cc:1367] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.015s
I20240501 13:56:22.698381 24916 catalog_manager.cc:5484] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:22.699084 25207 tablet_bootstrap.cc:492] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:22.699543 25207 ts_tablet_manager.cc:1367] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.000s
I20240501 13:56:22.699818 25409 leader_election.cc:288] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Requested vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.694968 25210 consensus_queue.cc:260] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.701892 25210 ts_tablet_manager.cc:1398] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20240501 13:56:22.702128 25207 raft_consensus.cc:348] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.702713 25207 raft_consensus.cc:374] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.702899 25207 raft_consensus.cc:729] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.703498 25207 consensus_queue.cc:260] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.701099 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b085a215235d482e86b0a9529458b3c6" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:22.705890 25093 raft_consensus.cc:3037] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.707661 25209 raft_consensus.cc:348] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.708241 25209 raft_consensus.cc:374] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.708498 25209 raft_consensus.cc:729] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.708461 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b085a215235d482e86b0a9529458b3c6" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:22.705560 25207 ts_tablet_manager.cc:1398] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240501 13:56:22.709522 25209 consensus_queue.cc:260] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.710553 25159 raft_consensus.cc:3037] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.711089 25207 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15: Bootstrap starting.
I20240501 13:56:22.712082 25209 ts_tablet_manager.cc:1398] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240501 13:56:22.717361 25414 raft_consensus.cc:482] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.717895 25414 raft_consensus.cc:504] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.721820 25414 leader_election.cc:288] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.723481 25209 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06: Bootstrap starting.
I20240501 13:56:22.725090 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
W20240501 13:56:22.726330 25058 leader_election.cc:341] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.728919 25207 tablet_bootstrap.cc:654] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.730423 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.732051 25414 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240501 13:56:22.747880 25011 raft_consensus.cc:2454] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 0.
I20240501 13:56:22.750481 25059 leader_election.cc:302] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.751839 25537 raft_consensus.cc:2781] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.752218 25537 raft_consensus.cc:482] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.752597 25537 raft_consensus.cc:3037] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.760274 25159 raft_consensus.cc:2454] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:22.766340 25209 tablet_bootstrap.cc:654] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06: Neither blocks nor log segments found. Creating new log.
I20240501 13:56:22.769338 25093 raft_consensus.cc:2454] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 346b2bb141744d459f1fa3ad602d1a15 in term 1.
I20240501 13:56:22.771672 25574 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:22.772020 24982 leader_election.cc:302] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:22.773875 25594 raft_consensus.cc:2781] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.774420 25594 raft_consensus.cc:686] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Becoming Leader. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Running, Role: LEADER
I20240501 13:56:22.775228 25594 consensus_queue.cc:237] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.780800 25563 raft_consensus.cc:482] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.781389 25563 raft_consensus.cc:504] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.785244 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:22.785871 25092 raft_consensus.cc:2454] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:22.788640 25537 raft_consensus.cc:504] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.793422 25131 leader_election.cc:302] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:22.794829 25561 raft_consensus.cc:2781] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.795276 25561 raft_consensus.cc:482] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.795650 25561 raft_consensus.cc:3037] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.797222 25207 tablet_bootstrap.cc:492] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15: No bootstrap required, opened a new log
I20240501 13:56:22.797724 25207 ts_tablet_manager.cc:1367] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent bootstrapping tablet: real 0.087s user 0.024s sys 0.018s
I20240501 13:56:22.800274 25011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.800971 25011 raft_consensus.cc:3037] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.802395 25563 leader_election.cc:288] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:22.802047 25207 raft_consensus.cc:348] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.804925 25537 leader_election.cc:288] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.805408 25207 raft_consensus.cc:374] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.805440 25161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:22.813663 25207 raft_consensus.cc:729] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346b2bb141744d459f1fa3ad602d1a15, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.814303 25207 consensus_queue.cc:260] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:22.815073 25058 leader_election.cc:341] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:56:22.813020 25011 raft_consensus.cc:2454] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:22.817221 25059 leader_election.cc:302] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters: ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.817459 25207 ts_tablet_manager.cc:1398] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20240501 13:56:22.818501 25537 raft_consensus.cc:2781] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.818894 25537 raft_consensus.cc:686] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:22.819761 25537 consensus_queue.cc:237] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.825361 25561 raft_consensus.cc:504] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.827596 25561 leader_election.cc:288] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:22.830461 25093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:22.831101 25093 raft_consensus.cc:3037] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.836760 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.837618 25018 raft_consensus.cc:2454] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:22.838536 25093 raft_consensus.cc:2454] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:22.840970 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098d5f02b04442d2acdc15fcb9a7b429" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.841871 25016 raft_consensus.cc:3037] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.851521 25131 leader_election.cc:302] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:22.858559 25209 tablet_bootstrap.cc:492] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06: No bootstrap required, opened a new log
I20240501 13:56:22.859066 25209 ts_tablet_manager.cc:1367] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06: Time spent bootstrapping tablet: real 0.136s user 0.010s sys 0.000s
I20240501 13:56:22.862565 25016 raft_consensus.cc:2454] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:22.877348 24915 catalog_manager.cc:5484] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 1, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:22.877570 25209 raft_consensus.cc:348] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.879518 25209 raft_consensus.cc:374] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:56:22.880264 25209 raft_consensus.cc:729] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef020bad8d24499281f69de00e90af06, State: Initialized, Role: FOLLOWER
I20240501 13:56:22.881081 25209 consensus_queue.cc:260] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.885402 25563 raft_consensus.cc:2781] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.886080 25563 raft_consensus.cc:686] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:22.887172 25563 consensus_queue.cc:237] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.887578 24916 catalog_manager.cc:5484] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129). New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:22.893574 25561 consensus_queue.cc:1035] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:22.894785 25209 ts_tablet_manager.cc:1398] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20240501 13:56:22.912667 25565 raft_consensus.cc:482] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.913298 25565 raft_consensus.cc:504] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.915305 25565 leader_election.cc:288] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:22.917622 25088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:22.918327 25088 raft_consensus.cc:2454] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:22.920141 25131 leader_election.cc:302] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:22.923012 25565 raft_consensus.cc:2781] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:56:22.923383 25565 raft_consensus.cc:482] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:22.924428 25565 raft_consensus.cc:3037] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.939333 25565 raft_consensus.cc:504] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
W20240501 13:56:22.941470 25133 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.129:33309, user_credentials={real_user=slave}} blocked reactor thread for 69744.5us
I20240501 13:56:22.943966 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e"
I20240501 13:56:22.944149 25269 consensus_queue.cc:1035] T b6e1a6d9d3d9424eb9fb5a781214940e P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:22.944809 25092 raft_consensus.cc:3037] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.949812 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:22.950589 25013 raft_consensus.cc:2454] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 0.
I20240501 13:56:22.951618 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d3ad40b4e34f969bec20bee2bc0162" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:22.952217 25016 raft_consensus.cc:3037] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:22.955051 25092 raft_consensus.cc:2454] T 34d3ad40b4e34f969bec20bee2bc0162 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:22.958240 25131 leader_election.cc:302] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:22.959038 25563 raft_consensus.cc:2781] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:56:22.959519 25563 raft_consensus.cc:686] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Becoming Leader. State: Replica: ef020bad8d24499281f69de00e90af06, State: Running, Role: LEADER
I20240501 13:56:22.960702 25409 raft_consensus.cc:482] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.960333 25563 consensus_queue.cc:237] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.961184 25409 raft_consensus.cc:504] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.962870 25409 leader_election.cc:288] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:22.964679 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:22.968756 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f348136225c444e1a2ed5658a3306eb7" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:22.970016 25016 raft_consensus.cc:2454] T 34d3ad40b4e34f969bec20bee2bc0162 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef020bad8d24499281f69de00e90af06 in term 1.
I20240501 13:56:22.968011 24916 catalog_manager.cc:5484] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:22.972100 24983 leader_election.cc:302] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:22.972826 25594 raft_consensus.cc:2726] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:22.951678 25565 leader_election.cc:288] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 1 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:22.991982 24916 catalog_manager.cc:5484] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef020bad8d24499281f69de00e90af06 (127.23.70.131). New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } }
I20240501 13:56:22.996471 25594 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:22.997069 25594 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:22.999047 25594 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:23.000691 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:23.001189 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "346b2bb141744d459f1fa3ad602d1a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:23.001513 25092 raft_consensus.cc:2379] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate 171c5cc5f6df4820bc16946f8717737e in this term.
I20240501 13:56:23.001874 25159 raft_consensus.cc:2379] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 346b2bb141744d459f1fa3ad602d1a15 in current term 1: Already voted for candidate ef020bad8d24499281f69de00e90af06 in this term.
I20240501 13:56:23.002844 24983 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 346b2bb141744d459f1fa3ad602d1a15; no voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06
I20240501 13:56:23.003554 25594 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:23.010566 25594 raft_consensus.cc:2726] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:56:23.192423 25574 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 13:56:23.199579 24964 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 13:56:23.219133 25574 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.232204 25016 raft_consensus.cc:3037] T f348136225c444e1a2ed5658a3306eb7 P 346b2bb141744d459f1fa3ad602d1a15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:23.255247 25537 consensus_queue.cc:1035] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.265354 25159 raft_consensus.cc:3037] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:56:23.276631 25565 raft_consensus.cc:482] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:23.277161 25565 raft_consensus.cc:504] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:23.281301 25016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:23.282361 25533 consensus_queue.cc:1035] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.298463 25092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8ee2e10bbd483e9356972bab5ea084" candidate_uuid: "ef020bad8d24499281f69de00e90af06" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171c5cc5f6df4820bc16946f8717737e" is_pre_election: true
I20240501 13:56:23.300047 25131 leader_election.cc:302] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef020bad8d24499281f69de00e90af06; no voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15
I20240501 13:56:23.301510 25269 raft_consensus.cc:2726] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 13:56:23.308708 25565 leader_election.cc:288] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:23.333438 25585 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:23.348764 25409 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.363201 25565 consensus_queue.cc:1035] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.373576 25269 consensus_queue.cc:1035] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.418123 25269 consensus_queue.cc:1035] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.431219 25563 consensus_queue.cc:1035] T 34d3ad40b4e34f969bec20bee2bc0162 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:23.837327 25537 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:56:23.837757 25537 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:23.839219 25537 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:23.840180 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06" is_pre_election: true
I20240501 13:56:23.840183 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15" is_pre_election: true
I20240501 13:56:23.840950 25159 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:23.841068 25013 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 1.
I20240501 13:56:23.842073 25058 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, ef020bad8d24499281f69de00e90af06; no voters:
I20240501 13:56:23.842857 25537 raft_consensus.cc:2781] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:56:23.843242 25537 raft_consensus.cc:482] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:56:23.843575 25537 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:23.850461 25537 raft_consensus.cc:504] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:23.853909 25537 leader_election.cc:288] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 2 election: Requested vote from peers 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309), ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:23.853951 25013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346b2bb141744d459f1fa3ad602d1a15"
I20240501 13:56:23.855028 25013 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:23.858641 25159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8b602c225b410792a822c1be9b69e9" candidate_uuid: "171c5cc5f6df4820bc16946f8717737e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef020bad8d24499281f69de00e90af06"
I20240501 13:56:23.859426 25159 raft_consensus.cc:3037] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:56:23.871033 25013 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 2.
I20240501 13:56:23.870771 25159 raft_consensus.cc:2454] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171c5cc5f6df4820bc16946f8717737e in term 2.
I20240501 13:56:23.872081 25059 leader_election.cc:302] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171c5cc5f6df4820bc16946f8717737e, 346b2bb141744d459f1fa3ad602d1a15; no voters:
I20240501 13:56:23.872857 25533 raft_consensus.cc:2781] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:56:23.873256 25533 raft_consensus.cc:686] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [term 2 LEADER]: Becoming Leader. State: Replica: 171c5cc5f6df4820bc16946f8717737e, State: Running, Role: LEADER
I20240501 13:56:23.874043 25533 consensus_queue.cc:237] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } }
I20240501 13:56:23.884930 24916 catalog_manager.cc:5484] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: term changed from 0 to 2, leader changed from <none> to 171c5cc5f6df4820bc16946f8717737e (127.23.70.130). New cstate: current_term: 2 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:24.320290 25013 raft_consensus.cc:1261] T 6f8b602c225b410792a822c1be9b69e9 P 346b2bb141744d459f1fa3ad602d1a15 [term 2 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 13:56:24.321370 25533 consensus_queue.cc:1035] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:56:24.330541 25159 raft_consensus.cc:1261] T 6f8b602c225b410792a822c1be9b69e9 P ef020bad8d24499281f69de00e90af06 [term 2 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 13:56:24.335600 25574 consensus_queue.cc:1035] T 6f8b602c225b410792a822c1be9b69e9 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 13:56:25.595710 23834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:56:25.631704 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:25.631704 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:56:25.632298 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:56:25.633088 23834 server_base.cc:1034] running on GCE node
I20240501 13:56:25.634415 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:56:25.634716 23834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:56:25.634969 23834 hybrid_clock.cc:648] HybridClock initialized: now 1714571785634948 us; error 0 us; skew 500 ppm
I20240501 13:56:25.635803 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:56:25.638913 23834 webserver.cc:460] Webserver started at http://127.23.70.132:39031/ using document root <none> and password file <none>
I20240501 13:56:25.639469 23834 fs_manager.cc:359] Metadata directory not provided
I20240501 13:56:25.639750 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:56:25.640067 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:56:25.641534 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-3-root/instance:
uuid: "dc721447aaca4b9ea5b345b35cf0e19e"
format_stamp: "Formatted at 2024-05-01 13:56:25 on dist-test-slave-s72h"
I20240501 13:56:25.646956 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 13:56:25.650553 25674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:25.651368 23834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 13:56:25.651718 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-3-root
uuid: "dc721447aaca4b9ea5b345b35cf0e19e"
format_stamp: "Formatted at 2024-05-01 13:56:25 on dist-test-slave-s72h"
I20240501 13:56:25.652009 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:56:25.672034 23834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:56:25.673413 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:56:25.674918 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:56:25.677711 23834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:56:25.677976 23834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:25.678295 23834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:56:25.678534 23834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:56:25.743430 23834 rpc_server.cc:307] RPC server started. Bound to: 127.23.70.132:38635
I20240501 13:56:25.743541 25736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.70.132:38635 every 8 connection(s)
I20240501 13:56:25.748009 23834 auto_leader_rebalancer-test.cc:405] add a tserver: dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:25.762082 25737 heartbeater.cc:347] Connected to a master server at 127.23.70.190:46015
I20240501 13:56:25.763237 25737 heartbeater.cc:464] Registering TS with master...
I20240501 13:56:25.764179 25737 heartbeater.cc:511] Master 127.23.70.190:46015 requested a full tablet report, sending...
I20240501 13:56:25.766297 24913 ts_manager.cc:194] Registered new tserver with Master: dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635)
I20240501 13:56:25.768047 24913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57578
I20240501 13:56:26.375605 25159 consensus_queue.cc:237] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.383066 25013 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.384264 25086 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.385226 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.386868 25563 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.399860 25269 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
W20240501 13:56:26.402936 25132 consensus_peers.cc:485] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3e4ec10c6ef41a9815b71da267be33f. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:26.405905 25086 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.404484 25013 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.443502 24913 catalog_manager.cc:5484] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:26.451575 25161 consensus_queue.cc:237] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.466351 25086 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.467782 25010 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.467839 25741 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 13:56:26.466401 25132 consensus_peers.cc:485] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e059d8fd99574a5eac0b49e9405d7bd4. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:26.469286 25751 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.482939 25741 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.485100 25010 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.490099 25086 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.510746 24916 catalog_manager.cc:5484] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.549656 25086 consensus_queue.cc:237] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.561303 25161 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.563647 25537 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.573832 25010 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:26.578102 25058 consensus_peers.cc:485] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 872135f76cb04f6ebfce1899ed32a5a0. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:26.578572 25759 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.582502 25759 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.584712 25161 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.590584 25013 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.604773 24916 catalog_manager.cc:5484] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.625406 25013 consensus_queue.cc:237] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.642470 25093 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.644762 25743 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.650730 25161 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:26.654975 25743 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:26.657487 25585 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.659312 25093 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
W20240501 13:56:26.678519 24983 consensus_peers.cc:485] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b7e26b87b054270bc41f6e57ff69ac0. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:26.665705 25161 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.682018 24916 catalog_manager.cc:5484] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:26.770628 25737 heartbeater.cc:503] Master 127.23.70.190:46015 was elected leader, sending a full tablet report...
I20240501 13:56:26.780583 25771 ts_tablet_manager.cc:902] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:26.797760 25771 tablet_copy_client.cc:308] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.131:40317
I20240501 13:56:26.815901 25177 tablet_copy_service.cc:140] P ef020bad8d24499281f69de00e90af06: Received BeginTabletCopySession request for tablet a3e4ec10c6ef41a9815b71da267be33f from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:36092)
I20240501 13:56:26.816622 25177 tablet_copy_service.cc:161] P ef020bad8d24499281f69de00e90af06: Beginning new tablet copy session on tablet a3e4ec10c6ef41a9815b71da267be33f from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:36092: session id = dc721447aaca4b9ea5b345b35cf0e19e-a3e4ec10c6ef41a9815b71da267be33f
I20240501 13:56:26.823170 25177 tablet_copy_source_session.cc:205] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:26.827239 25771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e4ec10c6ef41a9815b71da267be33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:26.858733 25771 tablet_copy_client.cc:783] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:26.859326 25771 tablet_copy_client.cc:647] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:26.863441 25771 tablet_copy_client.cc:523] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:26.872462 25771 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:26.892228 25771 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:26.893261 25771 tablet_bootstrap.cc:492] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:26.893949 25771 ts_tablet_manager.cc:1367] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.005s
I20240501 13:56:26.896546 25771 raft_consensus.cc:348] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.897379 25771 raft_consensus.cc:729] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:26.898003 25771 consensus_queue.cc:260] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:26.902065 25771 ts_tablet_manager.cc:1398] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240501 13:56:26.906105 25177 tablet_copy_service.cc:342] P ef020bad8d24499281f69de00e90af06: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-a3e4ec10c6ef41a9815b71da267be33f received from {username='slave'} at 127.0.0.1:36092
I20240501 13:56:26.906601 25177 tablet_copy_service.cc:433] P ef020bad8d24499281f69de00e90af06: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-a3e4ec10c6ef41a9815b71da267be33f on tablet a3e4ec10c6ef41a9815b71da267be33f with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:26.998973 25771 ts_tablet_manager.cc:902] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:27.017560 25771 tablet_copy_client.cc:308] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.131:40317
I20240501 13:56:27.019193 25177 tablet_copy_service.cc:140] P ef020bad8d24499281f69de00e90af06: Received BeginTabletCopySession request for tablet e059d8fd99574a5eac0b49e9405d7bd4 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:36092)
I20240501 13:56:27.019754 25177 tablet_copy_service.cc:161] P ef020bad8d24499281f69de00e90af06: Beginning new tablet copy session on tablet e059d8fd99574a5eac0b49e9405d7bd4 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:36092: session id = dc721447aaca4b9ea5b345b35cf0e19e-e059d8fd99574a5eac0b49e9405d7bd4
I20240501 13:56:27.025889 25177 tablet_copy_source_session.cc:205] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:27.028841 25771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e059d8fd99574a5eac0b49e9405d7bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:27.045010 25771 tablet_copy_client.cc:783] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:27.045596 25771 tablet_copy_client.cc:647] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:27.052646 25771 tablet_copy_client.cc:523] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:27.062247 25771 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:27.088424 25771 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:27.089309 25771 tablet_bootstrap.cc:492] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:27.089987 25771 ts_tablet_manager.cc:1367] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20240501 13:56:27.091979 25771 raft_consensus.cc:348] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.093053 25771 raft_consensus.cc:729] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:27.093725 25771 consensus_queue.cc:260] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.095682 25771 ts_tablet_manager.cc:1398] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 13:56:27.097265 25177 tablet_copy_service.cc:342] P ef020bad8d24499281f69de00e90af06: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-e059d8fd99574a5eac0b49e9405d7bd4 received from {username='slave'} at 127.0.0.1:36092
I20240501 13:56:27.097685 25177 tablet_copy_service.cc:433] P ef020bad8d24499281f69de00e90af06: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-e059d8fd99574a5eac0b49e9405d7bd4 on tablet e059d8fd99574a5eac0b49e9405d7bd4 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:27.165814 25771 ts_tablet_manager.cc:902] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309)
I20240501 13:56:27.185117 25771 tablet_copy_client.cc:308] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.129:33309
I20240501 13:56:27.190923 25788 ts_tablet_manager.cc:902] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:27.195226 25788 tablet_copy_client.cc:308] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.130:34553
I20240501 13:56:27.200744 25028 tablet_copy_service.cc:140] P 346b2bb141744d459f1fa3ad602d1a15: Received BeginTabletCopySession request for tablet 1b7e26b87b054270bc41f6e57ff69ac0 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:53980)
I20240501 13:56:27.201401 25028 tablet_copy_service.cc:161] P 346b2bb141744d459f1fa3ad602d1a15: Beginning new tablet copy session on tablet 1b7e26b87b054270bc41f6e57ff69ac0 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:53980: session id = dc721447aaca4b9ea5b345b35cf0e19e-1b7e26b87b054270bc41f6e57ff69ac0
I20240501 13:56:27.209401 25028 tablet_copy_source_session.cc:205] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:27.211982 25771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7e26b87b054270bc41f6e57ff69ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:27.218433 25103 tablet_copy_service.cc:140] P 171c5cc5f6df4820bc16946f8717737e: Received BeginTabletCopySession request for tablet 872135f76cb04f6ebfce1899ed32a5a0 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:34050)
I20240501 13:56:27.219079 25103 tablet_copy_service.cc:161] P 171c5cc5f6df4820bc16946f8717737e: Beginning new tablet copy session on tablet 872135f76cb04f6ebfce1899ed32a5a0 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:34050: session id = dc721447aaca4b9ea5b345b35cf0e19e-872135f76cb04f6ebfce1899ed32a5a0
I20240501 13:56:27.231846 25771 tablet_copy_client.cc:783] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:27.232491 25771 tablet_copy_client.cc:647] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:27.234766 25103 tablet_copy_source_session.cc:205] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:27.242496 25771 tablet_copy_client.cc:523] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:27.245774 25788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 872135f76cb04f6ebfce1899ed32a5a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:27.251272 25771 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:27.264830 25788 tablet_copy_client.cc:783] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:27.265676 25788 tablet_copy_client.cc:647] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:27.269328 25788 tablet_copy_client.cc:523] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:27.273005 25771 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:27.273901 25771 tablet_bootstrap.cc:492] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:27.274545 25771 ts_tablet_manager.cc:1367] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.008s
I20240501 13:56:27.277041 25771 raft_consensus.cc:348] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.277861 25771 raft_consensus.cc:729] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:27.278340 25788 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:27.278445 25771 consensus_queue.cc:260] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.280705 25771 ts_tablet_manager.cc:1398] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240501 13:56:27.282413 25028 tablet_copy_service.cc:342] P 346b2bb141744d459f1fa3ad602d1a15: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-1b7e26b87b054270bc41f6e57ff69ac0 received from {username='slave'} at 127.0.0.1:53980
I20240501 13:56:27.282892 25028 tablet_copy_service.cc:433] P 346b2bb141744d459f1fa3ad602d1a15: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-1b7e26b87b054270bc41f6e57ff69ac0 on tablet 1b7e26b87b054270bc41f6e57ff69ac0 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:27.306537 25788 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:27.307265 25788 tablet_bootstrap.cc:492] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:27.307884 25788 ts_tablet_manager.cc:1367] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.000s
I20240501 13:56:27.310269 25788 raft_consensus.cc:348] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.311065 25788 raft_consensus.cc:729] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:27.311559 25788 consensus_queue.cc:260] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:27.313812 25788 ts_tablet_manager.cc:1398] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20240501 13:56:27.315279 25103 tablet_copy_service.cc:342] P 171c5cc5f6df4820bc16946f8717737e: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-872135f76cb04f6ebfce1899ed32a5a0 received from {username='slave'} at 127.0.0.1:34050
I20240501 13:56:27.315673 25103 tablet_copy_service.cc:433] P 171c5cc5f6df4820bc16946f8717737e: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-872135f76cb04f6ebfce1899ed32a5a0 on tablet 872135f76cb04f6ebfce1899ed32a5a0 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:27.350783 25712 raft_consensus.cc:1203] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:27.424613 25712 raft_consensus.cc:1203] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:27.636322 25712 raft_consensus.cc:1203] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:27.665351 25712 raft_consensus.cc:1203] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:27.851747 25751 raft_consensus.cc:1050] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:27.854548 25751 consensus_queue.cc:237] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:27.869728 25712 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:27.871486 25741 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:27.874747 25093 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:27.880730 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:27.883180 25010 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:27.887944 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:27.893479 25086 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.894080 25269 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.912324 25712 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.914530 24915 catalog_manager.cc:5484] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.921571 25010 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.958905 25165 consensus_queue.cc:237] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:27.963832 25086 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:27.965096 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:27.967900 25712 raft_consensus.cc:1261] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:27.969446 25269 consensus_queue.cc:1035] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:27.973040 25751 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.975339 25092 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.979804 25712 raft_consensus.cc:2932] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:27.987586 24902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3e4ec10c6ef41a9815b71da267be33f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:27.994529 24916 catalog_manager.cc:5484] T a3e4ec10c6ef41a9815b71da267be33f P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.006067 25002 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e4ec10c6ef41a9815b71da267be33f with delete_type TABLET_DATA_TOMBSTONED (TS 346b2bb141744d459f1fa3ad602d1a15 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41354
I20240501 13:56:28.012562 25808 tablet_replica.cc:330] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: stopping tablet replica
I20240501 13:56:28.013538 25808 raft_consensus.cc:2229] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:28.014108 25808 raft_consensus.cc:2258] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:28.017642 25808 ts_tablet_manager.cc:1875] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:28.030838 25808 ts_tablet_manager.cc:1888] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:28.031301 25808 log.cc:1196] T a3e4ec10c6ef41a9815b71da267be33f P 346b2bb141744d459f1fa3ad602d1a15: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/wals/a3e4ec10c6ef41a9815b71da267be33f
I20240501 13:56:28.032896 24903 catalog_manager.cc:4830] TS 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): tablet a3e4ec10c6ef41a9815b71da267be33f (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:28.037494 25763 raft_consensus.cc:1050] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:28.039650 25763 consensus_queue.cc:237] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.046411 25585 raft_consensus.cc:1050] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:28.048123 25013 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.048280 25161 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.049544 25742 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:28.051599 25712 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.052070 25751 raft_consensus.cc:1050] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:28.059382 25751 consensus_queue.cc:237] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.056041 25585 consensus_queue.cc:237] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.075031 25712 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.055086 25763 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240501 13:56:28.076587 25741 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:28.079514 25086 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.080950 25784 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:28.080993 25741 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240501 13:56:28.087745 25016 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.096274 25269 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:28.108819 25269 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.116240 25093 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.118742 25742 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.120863 25743 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:28.120519 25010 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.124617 25711 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.137012 25162 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.137894 25013 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.141804 25710 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:28.142192 25821 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:28.144997 25799 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:28.148718 25092 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.149035 24913 catalog_manager.cc:5484] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:56:28.161859 24914 catalog_manager.cc:5484] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:56:28.162281 25712 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.177517 25159 consensus_queue.cc:237] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.179955 25092 consensus_queue.cc:237] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.190310 25711 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.193815 25741 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:28.211791 25011 raft_consensus.cc:1261] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.212430 25741 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.217094 25013 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.218869 25751 consensus_queue.cc:1035] T e059d8fd99574a5eac0b49e9405d7bd4 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:28.222855 25784 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:28.227432 25711 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.231899 25709 raft_consensus.cc:1261] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.235180 25811 consensus_queue.cc:1035] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:28.241329 25811 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.241616 25011 raft_consensus.cc:2932] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.243057 25010 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.263936 25585 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.270485 25709 raft_consensus.cc:2932] T 872135f76cb04f6ebfce1899ed32a5a0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.278339 25086 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.285398 24902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e059d8fd99574a5eac0b49e9405d7bd4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:28.284778 25711 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.295696 24901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 872135f76cb04f6ebfce1899ed32a5a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:28.295621 25163 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.302645 24915 catalog_manager.cc:5484] T e059d8fd99574a5eac0b49e9405d7bd4 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.304253 24915 catalog_manager.cc:5484] T 872135f76cb04f6ebfce1899ed32a5a0 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.324023 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet e059d8fd99574a5eac0b49e9405d7bd4 with delete_type TABLET_DATA_TOMBSTONED (TS 171c5cc5f6df4820bc16946f8717737e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33652
I20240501 13:56:28.327658 24914 catalog_manager.cc:5484] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.345489 25154 tablet_service.cc:1514] Processing DeleteTablet for tablet 872135f76cb04f6ebfce1899ed32a5a0 with delete_type TABLET_DATA_TOMBSTONED (TS ef020bad8d24499281f69de00e90af06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40490
I20240501 13:56:28.349681 25827 tablet_replica.cc:330] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: stopping tablet replica
I20240501 13:56:28.350858 25827 raft_consensus.cc:2229] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:28.351495 25827 raft_consensus.cc:2258] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:28.359030 25828 tablet_replica.cc:330] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: stopping tablet replica
I20240501 13:56:28.360096 25828 raft_consensus.cc:2229] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:28.360551 25828 pending_rounds.cc:62] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Trying to abort 1 pending ops.
I20240501 13:56:28.357972 25827 ts_tablet_manager.cc:1875] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:28.360821 25828 pending_rounds.cc:68] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022886043805552640 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "872135f76cb04f6ebfce1899ed32a5a0" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } } }
I20240501 13:56:28.362391 25828 raft_consensus.cc:2894] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 13:56:28.362814 25828 raft_consensus.cc:2866] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 13:56:28.363174 25828 raft_consensus.cc:2258] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:28.367183 25828 ts_tablet_manager.cc:1875] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:28.375866 25010 consensus_queue.cc:237] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:28.382102 25827 ts_tablet_manager.cc:1888] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:28.382588 25827 log.cc:1196] T e059d8fd99574a5eac0b49e9405d7bd4 P 171c5cc5f6df4820bc16946f8717737e: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/wals/e059d8fd99574a5eac0b49e9405d7bd4
I20240501 13:56:28.387284 24901 catalog_manager.cc:4830] TS 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): tablet e059d8fd99574a5eac0b49e9405d7bd4 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:28.387768 25828 ts_tablet_manager.cc:1888] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:28.388214 25828 log.cc:1196] T 872135f76cb04f6ebfce1899ed32a5a0 P ef020bad8d24499281f69de00e90af06: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/wals/872135f76cb04f6ebfce1899ed32a5a0
I20240501 13:56:28.389979 24902 catalog_manager.cc:4830] TS ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): tablet 872135f76cb04f6ebfce1899ed32a5a0 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:28.391391 25711 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.391521 25161 raft_consensus.cc:1261] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:28.395241 25821 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:28.396237 25585 consensus_queue.cc:1035] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:28.405565 25585 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.406896 25711 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.409400 25161 raft_consensus.cc:2932] T 1b7e26b87b054270bc41f6e57ff69ac0 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.418275 24903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b7e26b87b054270bc41f6e57ff69ac0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:28.418614 24913 catalog_manager.cc:5484] T 1b7e26b87b054270bc41f6e57ff69ac0 P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:28.430984 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7e26b87b054270bc41f6e57ff69ac0 with delete_type TABLET_DATA_TOMBSTONED (TS 171c5cc5f6df4820bc16946f8717737e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33652
I20240501 13:56:28.431816 25827 tablet_replica.cc:330] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: stopping tablet replica
I20240501 13:56:28.432761 25827 raft_consensus.cc:2229] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:28.433290 25827 raft_consensus.cc:2258] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:28.435773 25827 ts_tablet_manager.cc:1875] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:28.450822 25827 ts_tablet_manager.cc:1888] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:28.451221 25827 log.cc:1196] T 1b7e26b87b054270bc41f6e57ff69ac0 P 171c5cc5f6df4820bc16946f8717737e: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/wals/1b7e26b87b054270bc41f6e57ff69ac0
I20240501 13:56:28.452533 24901 catalog_manager.cc:4830] TS 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): tablet 1b7e26b87b054270bc41f6e57ff69ac0 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:29.489827 25086 consensus_queue.cc:237] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
W20240501 13:56:29.495613 25058 consensus_peers.cc:485] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b683a5a430741e6ac1949303d905584. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:29.495880 25161 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.497197 25784 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:29.497896 25013 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.499384 25829 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.505548 25784 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.516558 25161 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.525154 24914 catalog_manager.cc:5484] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:29.525346 25011 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.524305 25163 consensus_queue.cc:237] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
W20240501 13:56:29.536238 25132 consensus_peers.cc:485] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b159610f514496a8a1faf7d78c6c839. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:29.541746 25086 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.543269 25751 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.554636 25013 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.556205 25751 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.568570 25269 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.577306 25088 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.609720 25010 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.619975 25013 consensus_queue.cc:237] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
W20240501 13:56:29.627724 24983 consensus_peers.cc:485] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c77fc3f2c34f4b58bb2af4583495adba. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:29.630302 25088 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.631809 25585 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.632673 25161 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.636947 25585 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.644466 25585 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.642808 24913 catalog_manager.cc:5484] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.646476 25088 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.658318 25163 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.671346 24914 catalog_manager.cc:5484] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:29.672421 25161 consensus_queue.cc:237] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:29.681246 25010 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:29.681515 25088 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:29.682247 25132 consensus_peers.cc:485] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b14ca53436a04447b62816101d8b838b. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:29.683558 25841 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.685040 25269 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:29.693588 25269 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.696393 25088 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.699153 25010 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:29.716079 24915 catalog_manager.cc:5484] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:29.978615 25852 ts_tablet_manager.cc:902] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:29.983420 25852 tablet_copy_client.cc:308] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.130:34553
I20240501 13:56:29.985055 25103 tablet_copy_service.cc:140] P 171c5cc5f6df4820bc16946f8717737e: Received BeginTabletCopySession request for tablet 0b683a5a430741e6ac1949303d905584 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:34050)
I20240501 13:56:29.985507 25103 tablet_copy_service.cc:161] P 171c5cc5f6df4820bc16946f8717737e: Beginning new tablet copy session on tablet 0b683a5a430741e6ac1949303d905584 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:34050: session id = dc721447aaca4b9ea5b345b35cf0e19e-0b683a5a430741e6ac1949303d905584
I20240501 13:56:29.991559 25103 tablet_copy_source_session.cc:205] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:29.994046 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b683a5a430741e6ac1949303d905584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:30.007961 25852 tablet_copy_client.cc:783] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:30.008543 25852 tablet_copy_client.cc:647] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:30.012171 25852 tablet_copy_client.cc:523] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:30.021075 25852 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:30.053215 25852 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:30.054096 25852 tablet_bootstrap.cc:492] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:30.054773 25852 ts_tablet_manager.cc:1367] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.015s
I20240501 13:56:30.057410 25852 raft_consensus.cc:348] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.058370 25852 raft_consensus.cc:729] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:30.059032 25852 consensus_queue.cc:260] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.068881 25852 ts_tablet_manager.cc:1398] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20240501 13:56:30.079705 25103 tablet_copy_service.cc:342] P 171c5cc5f6df4820bc16946f8717737e: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-0b683a5a430741e6ac1949303d905584 received from {username='slave'} at 127.0.0.1:34050
I20240501 13:56:30.080163 25103 tablet_copy_service.cc:433] P 171c5cc5f6df4820bc16946f8717737e: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-0b683a5a430741e6ac1949303d905584 on tablet 0b683a5a430741e6ac1949303d905584 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:30.085634 25859 ts_tablet_manager.cc:902] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:30.097514 25859 tablet_copy_client.cc:308] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.131:40317
I20240501 13:56:30.099011 25177 tablet_copy_service.cc:140] P ef020bad8d24499281f69de00e90af06: Received BeginTabletCopySession request for tablet 7b159610f514496a8a1faf7d78c6c839 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:36092)
I20240501 13:56:30.099555 25177 tablet_copy_service.cc:161] P ef020bad8d24499281f69de00e90af06: Beginning new tablet copy session on tablet 7b159610f514496a8a1faf7d78c6c839 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:36092: session id = dc721447aaca4b9ea5b345b35cf0e19e-7b159610f514496a8a1faf7d78c6c839
I20240501 13:56:30.106978 25177 tablet_copy_source_session.cc:205] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:30.110165 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b159610f514496a8a1faf7d78c6c839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:30.119863 25852 ts_tablet_manager.cc:902] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309)
I20240501 13:56:30.126286 25859 tablet_copy_client.cc:783] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:30.126925 25859 tablet_copy_client.cc:647] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:30.139168 25859 tablet_copy_client.cc:523] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:30.154670 25859 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:30.176173 25852 tablet_copy_client.cc:308] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.129:33309
I20240501 13:56:30.179391 25028 tablet_copy_service.cc:140] P 346b2bb141744d459f1fa3ad602d1a15: Received BeginTabletCopySession request for tablet c77fc3f2c34f4b58bb2af4583495adba from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:53980)
I20240501 13:56:30.179927 25028 tablet_copy_service.cc:161] P 346b2bb141744d459f1fa3ad602d1a15: Beginning new tablet copy session on tablet c77fc3f2c34f4b58bb2af4583495adba from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:53980: session id = dc721447aaca4b9ea5b345b35cf0e19e-c77fc3f2c34f4b58bb2af4583495adba
I20240501 13:56:30.184041 25859 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:30.184821 25859 tablet_bootstrap.cc:492] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:30.185314 25859 ts_tablet_manager.cc:1367] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.003s
I20240501 13:56:30.185755 25028 tablet_copy_source_session.cc:205] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:30.188609 25852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c77fc3f2c34f4b58bb2af4583495adba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:30.188654 25859 raft_consensus.cc:348] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.190069 25859 raft_consensus.cc:729] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:30.190790 25859 consensus_queue.cc:260] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.193136 25859 ts_tablet_manager.cc:1398] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240501 13:56:30.196754 25177 tablet_copy_service.cc:342] P ef020bad8d24499281f69de00e90af06: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-7b159610f514496a8a1faf7d78c6c839 received from {username='slave'} at 127.0.0.1:36092
I20240501 13:56:30.197175 25177 tablet_copy_service.cc:433] P ef020bad8d24499281f69de00e90af06: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-7b159610f514496a8a1faf7d78c6c839 on tablet 7b159610f514496a8a1faf7d78c6c839 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:30.210561 25852 tablet_copy_client.cc:783] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:30.211143 25852 tablet_copy_client.cc:647] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:30.214797 25852 tablet_copy_client.cc:523] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:30.215965 25859 ts_tablet_manager.cc:902] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
I20240501 13:56:30.220119 25859 tablet_copy_client.cc:308] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.131:40317
I20240501 13:56:30.221474 25177 tablet_copy_service.cc:140] P ef020bad8d24499281f69de00e90af06: Received BeginTabletCopySession request for tablet b14ca53436a04447b62816101d8b838b from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:36092)
I20240501 13:56:30.221916 25177 tablet_copy_service.cc:161] P ef020bad8d24499281f69de00e90af06: Beginning new tablet copy session on tablet b14ca53436a04447b62816101d8b838b from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:36092: session id = dc721447aaca4b9ea5b345b35cf0e19e-b14ca53436a04447b62816101d8b838b
I20240501 13:56:30.225675 25852 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:30.232609 25177 tablet_copy_source_session.cc:205] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:30.235566 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14ca53436a04447b62816101d8b838b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:30.247901 25852 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:30.248898 25852 tablet_bootstrap.cc:492] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:30.249447 25852 ts_tablet_manager.cc:1367] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.006s
I20240501 13:56:30.252096 25852 raft_consensus.cc:348] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.253074 25852 raft_consensus.cc:729] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:30.253340 25859 tablet_copy_client.cc:783] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:30.254094 25859 tablet_copy_client.cc:647] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:30.253701 25852 consensus_queue.cc:260] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.256129 25852 ts_tablet_manager.cc:1398] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 13:56:30.258177 25028 tablet_copy_service.cc:342] P 346b2bb141744d459f1fa3ad602d1a15: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-c77fc3f2c34f4b58bb2af4583495adba received from {username='slave'} at 127.0.0.1:53980
I20240501 13:56:30.258584 25028 tablet_copy_service.cc:433] P 346b2bb141744d459f1fa3ad602d1a15: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-c77fc3f2c34f4b58bb2af4583495adba on tablet c77fc3f2c34f4b58bb2af4583495adba with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:30.259121 25859 tablet_copy_client.cc:523] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:30.270169 25859 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:30.290066 25859 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:30.291026 25859 tablet_bootstrap.cc:492] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:30.291731 25859 ts_tablet_manager.cc:1367] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20240501 13:56:30.294116 25859 raft_consensus.cc:348] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.295025 25859 raft_consensus.cc:729] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:30.295702 25859 consensus_queue.cc:260] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:30.298306 25859 ts_tablet_manager.cc:1398] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20240501 13:56:30.300319 25177 tablet_copy_service.cc:342] P ef020bad8d24499281f69de00e90af06: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-b14ca53436a04447b62816101d8b838b received from {username='slave'} at 127.0.0.1:36092
I20240501 13:56:30.300762 25177 tablet_copy_service.cc:433] P ef020bad8d24499281f69de00e90af06: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-b14ca53436a04447b62816101d8b838b on tablet b14ca53436a04447b62816101d8b838b with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:30.594178 25711 raft_consensus.cc:1203] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:30.649343 25711 raft_consensus.cc:1203] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:30.661586 25711 raft_consensus.cc:1203] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:30.687727 25711 raft_consensus.cc:1203] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:31.104854 25751 raft_consensus.cc:1050] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:31.106765 25751 consensus_queue.cc:237] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.112432 25711 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.114142 25088 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.114795 25751 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.117775 25016 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.120252 25269 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.126545 25841 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.151919 25751 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.166280 25711 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.171185 25016 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.175865 25876 raft_consensus.cc:1050] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:31.175421 24914 catalog_manager.cc:5484] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:56:31.176363 25088 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.178445 25876 consensus_queue.cc:237] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.209798 25010 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.211056 25712 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.211529 25886 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.215104 25877 raft_consensus.cc:1050] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:31.216984 25163 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.223234 25825 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:31.227082 25825 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:31.226967 25877 consensus_queue.cc:237] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.281250 25088 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.283471 25162 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.286537 25888 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 13:56:31.289559 25585 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.292830 25712 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.312291 25877 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:31.315338 25825 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.320477 25161 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.330973 25883 raft_consensus.cc:1050] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:31.333572 25883 consensus_queue.cc:237] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.345108 25016 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.343428 25709 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.361223 25092 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.361838 25011 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.363879 25883 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:31.410194 25751 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:31.420058 25712 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:31.421721 25751 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:31.428601 25163 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.426505 25898 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.474013 25711 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.541488 25162 consensus_queue.cc:237] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.578716 24915 catalog_manager.cc:5484] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 13:56:31.586961 25131 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 47344.1us
I20240501 13:56:31.592965 25711 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 13:56:31.607009 24982 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 63421.8us
I20240501 13:56:31.607475 25880 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.609364 25751 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.611055 25712 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.614159 25016 raft_consensus.cc:1261] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.618158 24913 catalog_manager.cc:5484] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.620071 25841 consensus_queue.cc:1035] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240501 13:56:31.623837 25269 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.623919 25092 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.631409 25088 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.637312 25086 consensus_queue.cc:237] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.631430 25711 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.643000 25011 raft_consensus.cc:2932] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.645262 25016 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.651670 24902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b14ca53436a04447b62816101d8b838b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:31.656406 25710 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.658519 25825 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.663110 25011 raft_consensus.cc:1261] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.664561 25886 consensus_queue.cc:1035] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.668737 25892 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
W20240501 13:56:31.679070 24902 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet b14ca53436a04447b62816101d8b838b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240501 13:56:31.682790 24914 catalog_manager.cc:5484] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.684741 24901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b683a5a430741e6ac1949303d905584 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:31.692811 25011 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.696815 25711 raft_consensus.cc:2932] T 0b683a5a430741e6ac1949303d905584 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.693585 24914 catalog_manager.cc:5484] T b14ca53436a04447b62816101d8b838b P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.714227 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet b14ca53436a04447b62816101d8b838b with delete_type TABLET_DATA_TOMBSTONED (TS 171c5cc5f6df4820bc16946f8717737e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33652
I20240501 13:56:31.715852 25011 consensus_queue.cc:237] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.724674 25088 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.726152 25877 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.733012 25711 raft_consensus.cc:1261] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.733945 25877 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.735139 25585 consensus_queue.cc:1035] T c77fc3f2c34f4b58bb2af4583495adba P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:31.735700 24914 catalog_manager.cc:5484] T 0b683a5a430741e6ac1949303d905584 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.740497 25088 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.745159 25163 consensus_queue.cc:237] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:31.740259 25711 raft_consensus.cc:2932] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.751106 25710 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.752422 25841 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.753504 25086 raft_consensus.cc:1261] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:31.754705 25269 consensus_queue.cc:1035] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:31.759416 25883 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.760998 25710 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.763407 25086 raft_consensus.cc:2932] T 7b159610f514496a8a1faf7d78c6c839 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.772522 24903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c77fc3f2c34f4b58bb2af4583495adba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:31.777808 24902 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b159610f514496a8a1faf7d78c6c839 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:31.789558 25154 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b683a5a430741e6ac1949303d905584 with delete_type TABLET_DATA_TOMBSTONED (TS ef020bad8d24499281f69de00e90af06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40490
I20240501 13:56:31.794093 25911 tablet_replica.cc:330] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: stopping tablet replica
I20240501 13:56:31.795150 25911 raft_consensus.cc:2229] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:31.795804 25911 raft_consensus.cc:2258] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:31.801578 25912 tablet_replica.cc:330] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: stopping tablet replica
I20240501 13:56:31.804587 24913 catalog_manager.cc:5484] T c77fc3f2c34f4b58bb2af4583495adba P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.806262 24913 catalog_manager.cc:5484] T 7b159610f514496a8a1faf7d78c6c839 P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:31.814852 25911 ts_tablet_manager.cc:1875] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:31.821611 25912 raft_consensus.cc:2229] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:31.822422 25912 raft_consensus.cc:2258] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:31.829532 25154 tablet_service.cc:1514] Processing DeleteTablet for tablet c77fc3f2c34f4b58bb2af4583495adba with delete_type TABLET_DATA_TOMBSTONED (TS ef020bad8d24499281f69de00e90af06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40490
I20240501 13:56:31.831703 25912 ts_tablet_manager.cc:1875] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:31.835018 25002 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b159610f514496a8a1faf7d78c6c839 with delete_type TABLET_DATA_TOMBSTONED (TS 346b2bb141744d459f1fa3ad602d1a15 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41354
I20240501 13:56:31.836961 25911 ts_tablet_manager.cc:1888] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:31.837422 25911 log.cc:1196] T b14ca53436a04447b62816101d8b838b P 171c5cc5f6df4820bc16946f8717737e: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/wals/b14ca53436a04447b62816101d8b838b
I20240501 13:56:31.841893 25913 tablet_replica.cc:330] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: stopping tablet replica
I20240501 13:56:31.842561 24901 catalog_manager.cc:4830] TS 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): tablet b14ca53436a04447b62816101d8b838b (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:31.854596 25912 ts_tablet_manager.cc:1888] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:31.855019 25912 log.cc:1196] T 0b683a5a430741e6ac1949303d905584 P ef020bad8d24499281f69de00e90af06: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/wals/0b683a5a430741e6ac1949303d905584
I20240501 13:56:31.857097 25912 tablet_replica.cc:330] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: stopping tablet replica
I20240501 13:56:31.857836 25912 raft_consensus.cc:2229] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:31.858407 25912 raft_consensus.cc:2258] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:31.858500 25913 raft_consensus.cc:2229] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:31.859239 25913 raft_consensus.cc:2258] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:31.862398 25912 ts_tablet_manager.cc:1875] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:31.862522 25913 ts_tablet_manager.cc:1875] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:31.891475 24902 catalog_manager.cc:4830] TS ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): tablet 0b683a5a430741e6ac1949303d905584 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:31.945273 25912 ts_tablet_manager.cc:1888] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:31.946146 25912 log.cc:1196] T c77fc3f2c34f4b58bb2af4583495adba P ef020bad8d24499281f69de00e90af06: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/wals/c77fc3f2c34f4b58bb2af4583495adba
I20240501 13:56:31.947003 25913 ts_tablet_manager.cc:1888] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:31.947543 25913 log.cc:1196] T 7b159610f514496a8a1faf7d78c6c839 P 346b2bb141744d459f1fa3ad602d1a15: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/wals/7b159610f514496a8a1faf7d78c6c839
I20240501 13:56:31.949313 24902 catalog_manager.cc:4830] TS ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): tablet c77fc3f2c34f4b58bb2af4583495adba (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:31.950726 24903 catalog_manager.cc:4830] TS 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): tablet 7b159610f514496a8a1faf7d78c6c839 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:32.883466 25088 consensus_queue.cc:237] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:32.895395 25163 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:32.895979 25058 consensus_peers.cc:485] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf58f127779f4ebd94273cd7fda138cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:32.897411 25905 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:32.899919 25011 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:32.902015 25917 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:32.907482 25905 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:32.913619 25163 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:32.923501 25011 consensus_queue.cc:237] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:32.933055 24916 catalog_manager.cc:5484] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:32.940882 25018 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:32.956386 25088 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:32.957760 25898 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 13:56:32.962972 24983 consensus_peers.cc:485] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b085a215235d482e86b0a9529458b3c6. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:32.965229 25163 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:32.968508 25898 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:32.983052 25909 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:32.986357 25086 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:32.997566 25163 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.029778 25086 consensus_queue.cc:237] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.032964 24913 catalog_manager.cc:5484] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.037932 25159 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:33.038846 25058 consensus_peers.cc:485] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56eceec85d2f42f5a0baf57bb03b45f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:33.044042 25920 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:33.050097 25011 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:33.051793 25917 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:33.069164 25897 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.071336 25018 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.082167 25159 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.090541 25011 consensus_queue.cc:237] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.106218 25159 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:33.106877 24916 catalog_manager.cc:5484] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
W20240501 13:56:33.109898 24983 consensus_peers.cc:485] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdc140167ab54483b3db40ba7a61937f. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:33.111181 25931 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:33.116559 25086 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:33.118191 25931 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:33.143925 25909 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.150532 25086 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.151096 25163 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:33.174836 24915 catalog_manager.cc:5484] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:33.326602 25940 ts_tablet_manager.cc:902] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:33.333150 25940 tablet_copy_client.cc:308] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.130:34553
I20240501 13:56:33.335289 25103 tablet_copy_service.cc:140] P 171c5cc5f6df4820bc16946f8717737e: Received BeginTabletCopySession request for tablet bf58f127779f4ebd94273cd7fda138cd from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:34050)
I20240501 13:56:33.335963 25103 tablet_copy_service.cc:161] P 171c5cc5f6df4820bc16946f8717737e: Beginning new tablet copy session on tablet bf58f127779f4ebd94273cd7fda138cd from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:34050: session id = dc721447aaca4b9ea5b345b35cf0e19e-bf58f127779f4ebd94273cd7fda138cd
I20240501 13:56:33.342619 25103 tablet_copy_source_session.cc:205] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:33.346524 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf58f127779f4ebd94273cd7fda138cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:33.360363 25940 tablet_copy_client.cc:783] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:33.360991 25940 tablet_copy_client.cc:647] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:33.364955 25940 tablet_copy_client.cc:523] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:33.374739 25940 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:33.393188 25940 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:33.393913 25940 tablet_bootstrap.cc:492] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:33.394488 25940 ts_tablet_manager.cc:1367] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20240501 13:56:33.397540 25940 raft_consensus.cc:348] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.398236 25940 raft_consensus.cc:729] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:33.398869 25940 consensus_queue.cc:260] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.403522 25940 ts_tablet_manager.cc:1398] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240501 13:56:33.405117 25103 tablet_copy_service.cc:342] P 171c5cc5f6df4820bc16946f8717737e: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-bf58f127779f4ebd94273cd7fda138cd received from {username='slave'} at 127.0.0.1:34050
I20240501 13:56:33.405558 25103 tablet_copy_service.cc:433] P 171c5cc5f6df4820bc16946f8717737e: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-bf58f127779f4ebd94273cd7fda138cd on tablet bf58f127779f4ebd94273cd7fda138cd with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:33.412779 25947 ts_tablet_manager.cc:902] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309)
I20240501 13:56:33.433091 25947 tablet_copy_client.cc:308] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.129:33309
I20240501 13:56:33.438691 25028 tablet_copy_service.cc:140] P 346b2bb141744d459f1fa3ad602d1a15: Received BeginTabletCopySession request for tablet b085a215235d482e86b0a9529458b3c6 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:53980)
I20240501 13:56:33.439276 25028 tablet_copy_service.cc:161] P 346b2bb141744d459f1fa3ad602d1a15: Beginning new tablet copy session on tablet b085a215235d482e86b0a9529458b3c6 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:53980: session id = dc721447aaca4b9ea5b345b35cf0e19e-b085a215235d482e86b0a9529458b3c6
I20240501 13:56:33.453860 25028 tablet_copy_source_session.cc:205] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:33.457093 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b085a215235d482e86b0a9529458b3c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:33.477952 25947 tablet_copy_client.cc:783] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:33.478516 25947 tablet_copy_client.cc:647] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:33.482697 25947 tablet_copy_client.cc:523] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:33.487030 25940 ts_tablet_manager.cc:902] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
I20240501 13:56:33.492240 25947 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:33.520485 25947 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:33.521081 25940 tablet_copy_client.cc:308] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.130:34553
I20240501 13:56:33.521522 25947 tablet_bootstrap.cc:492] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:33.522284 25947 ts_tablet_manager.cc:1367] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.008s
I20240501 13:56:33.523651 25103 tablet_copy_service.cc:140] P 171c5cc5f6df4820bc16946f8717737e: Received BeginTabletCopySession request for tablet 56eceec85d2f42f5a0baf57bb03b45f4 from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:34050)
I20240501 13:56:33.524204 25103 tablet_copy_service.cc:161] P 171c5cc5f6df4820bc16946f8717737e: Beginning new tablet copy session on tablet 56eceec85d2f42f5a0baf57bb03b45f4 from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:34050: session id = dc721447aaca4b9ea5b345b35cf0e19e-56eceec85d2f42f5a0baf57bb03b45f4
I20240501 13:56:33.525216 25947 raft_consensus.cc:348] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.526113 25947 raft_consensus.cc:729] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:33.526785 25947 consensus_queue.cc:260] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.531276 25947 ts_tablet_manager.cc:1398] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 13:56:33.537976 25028 tablet_copy_service.cc:342] P 346b2bb141744d459f1fa3ad602d1a15: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-b085a215235d482e86b0a9529458b3c6 received from {username='slave'} at 127.0.0.1:53980
I20240501 13:56:33.538334 25028 tablet_copy_service.cc:433] P 346b2bb141744d459f1fa3ad602d1a15: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-b085a215235d482e86b0a9529458b3c6 on tablet b085a215235d482e86b0a9529458b3c6 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:33.533170 25103 tablet_copy_source_session.cc:205] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:33.543452 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56eceec85d2f42f5a0baf57bb03b45f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:33.556527 25940 tablet_copy_client.cc:783] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:33.557097 25940 tablet_copy_client.cc:647] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:33.560933 25940 tablet_copy_client.cc:523] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:33.568004 25940 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:33.586637 25940 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:33.587479 25940 tablet_bootstrap.cc:492] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:33.588132 25940 ts_tablet_manager.cc:1367] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20240501 13:56:33.590890 25940 raft_consensus.cc:348] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.591768 25940 raft_consensus.cc:729] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:33.592424 25940 consensus_queue.cc:260] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.595196 25940 ts_tablet_manager.cc:1398] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 13:56:33.596536 25103 tablet_copy_service.cc:342] P 171c5cc5f6df4820bc16946f8717737e: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-56eceec85d2f42f5a0baf57bb03b45f4 received from {username='slave'} at 127.0.0.1:34050
I20240501 13:56:33.596964 25103 tablet_copy_service.cc:433] P 171c5cc5f6df4820bc16946f8717737e: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-56eceec85d2f42f5a0baf57bb03b45f4 on tablet 56eceec85d2f42f5a0baf57bb03b45f4 with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:33.665083 25940 ts_tablet_manager.cc:902] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309)
I20240501 13:56:33.685084 25940 tablet_copy_client.cc:308] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Beginning tablet copy session from remote peer at address 127.23.70.129:33309
I20240501 13:56:33.687242 25028 tablet_copy_service.cc:140] P 346b2bb141744d459f1fa3ad602d1a15: Received BeginTabletCopySession request for tablet bdc140167ab54483b3db40ba7a61937f from peer dc721447aaca4b9ea5b345b35cf0e19e ({username='slave'} at 127.0.0.1:53980)
I20240501 13:56:33.687815 25028 tablet_copy_service.cc:161] P 346b2bb141744d459f1fa3ad602d1a15: Beginning new tablet copy session on tablet bdc140167ab54483b3db40ba7a61937f from peer dc721447aaca4b9ea5b345b35cf0e19e at {username='slave'} at 127.0.0.1:53980: session id = dc721447aaca4b9ea5b345b35cf0e19e-bdc140167ab54483b3db40ba7a61937f
I20240501 13:56:33.695830 25028 tablet_copy_source_session.cc:205] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:56:33.698587 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc140167ab54483b3db40ba7a61937f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:56:33.717466 25940 tablet_copy_client.cc:783] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 0 data blocks...
I20240501 13:56:33.719765 25940 tablet_copy_client.cc:647] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Starting download of 1 WAL segments...
I20240501 13:56:33.726768 25940 tablet_copy_client.cc:523] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:56:33.736374 25940 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap starting.
I20240501 13:56:33.755260 25940 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:56:33.756069 25940 tablet_bootstrap.cc:492] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Bootstrap complete.
I20240501 13:56:33.756670 25940 ts_tablet_manager.cc:1367] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240501 13:56:33.758394 25940 raft_consensus.cc:348] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.759102 25940 raft_consensus.cc:729] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc721447aaca4b9ea5b345b35cf0e19e, State: Initialized, Role: LEARNER
I20240501 13:56:33.759552 25940 consensus_queue.cc:260] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:33.761466 25940 ts_tablet_manager.cc:1398] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 13:56:33.762884 25028 tablet_copy_service.cc:342] P 346b2bb141744d459f1fa3ad602d1a15: Request end of tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-bdc140167ab54483b3db40ba7a61937f received from {username='slave'} at 127.0.0.1:53980
I20240501 13:56:33.763291 25028 tablet_copy_service.cc:433] P 346b2bb141744d459f1fa3ad602d1a15: ending tablet copy session dc721447aaca4b9ea5b345b35cf0e19e-bdc140167ab54483b3db40ba7a61937f on tablet bdc140167ab54483b3db40ba7a61937f with peer dc721447aaca4b9ea5b345b35cf0e19e
I20240501 13:56:33.832001 25711 raft_consensus.cc:1203] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:33.888991 25711 raft_consensus.cc:1203] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 13:56:33.959584 25058 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.132:38635, user_credentials={real_user=slave}} blocked reactor thread for 68807.5us
I20240501 13:56:34.051379 25710 raft_consensus.cc:1203] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:34.062580 25710 raft_consensus.cc:1203] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:56:34.310720 25964 raft_consensus.cc:1050] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:34.313141 25964 consensus_queue.cc:237] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.319108 25163 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.319108 25710 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.321731 25013 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.321943 25964 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.322849 25920 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.324043 25897 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.337591 25920 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.340886 25931 raft_consensus.cc:1050] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:34.343075 25931 consensus_queue.cc:237] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.347517 25018 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.352797 25710 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.354954 25159 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.366973 24916 catalog_manager.cc:5484] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.396567 25710 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.396760 25159 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.398530 25088 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.399349 25909 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.400292 25963 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.403637 25968 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.444383 25968 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.447826 25092 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.453464 25162 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.477401 25710 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.492653 25088 consensus_queue.cc:237] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.494469 24913 catalog_manager.cc:5484] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.499543 25162 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.504369 25710 raft_consensus.cc:1261] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.504649 25897 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:34.506269 25920 consensus_queue.cc:1035] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:34.516384 25964 raft_consensus.cc:1050] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:34.516307 25162 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.519888 25018 consensus_queue.cc:237] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.520167 25964 consensus_queue.cc:237] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.528748 25710 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.514693 25897 raft_consensus.cc:2932] T bf58f127779f4ebd94273cd7fda138cd P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.544095 25712 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.546840 25086 raft_consensus.cc:1261] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.546694 25931 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:34.549865 24916 catalog_manager.cc:5484] T bf58f127779f4ebd94273cd7fda138cd P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.550844 25963 consensus_queue.cc:1035] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:34.560917 24901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bf58f127779f4ebd94273cd7fda138cd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:34.565897 25002 tablet_service.cc:1514] Processing DeleteTablet for tablet bf58f127779f4ebd94273cd7fda138cd with delete_type TABLET_DATA_TOMBSTONED (TS 346b2bb141744d459f1fa3ad602d1a15 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41354
I20240501 13:56:34.565172 25968 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.541939 25711 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.542085 25163 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.573951 25964 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:34.574625 25086 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.576265 25964 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.578090 25963 raft_consensus.cc:1050] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15: attempting to promote NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e to VOTER
I20240501 13:56:34.583487 25711 raft_consensus.cc:2932] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.585634 25963 consensus_queue.cc:237] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.606431 25011 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.610385 25973 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 13:56:34.646740 25964 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.652117 25011 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.658933 24903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b085a215235d482e86b0a9529458b3c6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:34.658924 25709 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.667012 24914 catalog_manager.cc:5484] T b085a215235d482e86b0a9529458b3c6 P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 3 to 4, VOTER ef020bad8d24499281f69de00e90af06 (127.23.70.131) evicted. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.688813 25983 tablet_replica.cc:330] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: stopping tablet replica
I20240501 13:56:34.694283 25711 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 LEARNER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.696044 25162 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.696380 25909 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.697852 25086 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:56:34.698995 25968 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:56:34.690100 25983 raft_consensus.cc:2229] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:34.706614 25162 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.705894 25931 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240501 13:56:34.723599 25983 raft_consensus.cc:2258] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:34.741940 24916 catalog_manager.cc:5484] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.771276 25154 tablet_service.cc:1514] Processing DeleteTablet for tablet b085a215235d482e86b0a9529458b3c6 with delete_type TABLET_DATA_TOMBSTONED (TS ef020bad8d24499281f69de00e90af06 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40490
W20240501 13:56:34.849628 25131 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.70.130:34553, user_credentials={real_user=slave}} blocked reactor thread for 76763.3us
I20240501 13:56:34.857810 25998 tablet_replica.cc:330] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: stopping tablet replica
I20240501 13:56:34.861044 25983 ts_tablet_manager.cc:1875] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:34.865826 25993 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.871116 25163 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.872885 25711 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.907418 24913 catalog_manager.cc:5484] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e reported cstate change: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.913914 25086 consensus_queue.cc:237] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.924574 25011 consensus_queue.cc:237] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } }
I20240501 13:56:34.925482 25086 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.929553 25998 raft_consensus.cc:2229] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:34.930442 25998 raft_consensus.cc:2258] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:34.938051 25998 ts_tablet_manager.cc:1875] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:34.944783 25159 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.944988 25711 raft_consensus.cc:1261] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 346b2bb141744d459f1fa3ad602d1a15: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.946470 25909 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:34.948885 25983 ts_tablet_manager.cc:1888] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:34.949388 25983 log.cc:1196] T bf58f127779f4ebd94273cd7fda138cd P 346b2bb141744d459f1fa3ad602d1a15: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/wals/bf58f127779f4ebd94273cd7fda138cd
I20240501 13:56:34.951565 25909 consensus_queue.cc:1035] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:56:34.952497 24903 catalog_manager.cc:4830] TS 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): tablet bf58f127779f4ebd94273cd7fda138cd (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:34.953505 25963 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.957132 25163 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.958469 25709 raft_consensus.cc:1261] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:56:34.957764 25711 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.958997 26003 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:34.973052 25987 consensus_queue.cc:1035] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 13:56:34.975390 24903 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bdc140167ab54483b3db40ba7a61937f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:34.971366 25159 raft_consensus.cc:2932] T bdc140167ab54483b3db40ba7a61937f P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.985496 26000 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.986796 25163 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:34.991631 24913 catalog_manager.cc:5484] T bdc140167ab54483b3db40ba7a61937f P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index 3 to 4, VOTER 171c5cc5f6df4820bc16946f8717737e (127.23.70.130) evicted. New cstate: current_term: 1 leader_uuid: "346b2bb141744d459f1fa3ad602d1a15" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:56:35.001227 25709 raft_consensus.cc:2932] T 56eceec85d2f42f5a0baf57bb03b45f4 P dc721447aaca4b9ea5b345b35cf0e19e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:35.005744 24901 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56eceec85d2f42f5a0baf57bb03b45f4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:56:35.018687 24915 catalog_manager.cc:5484] T 56eceec85d2f42f5a0baf57bb03b45f4 P ef020bad8d24499281f69de00e90af06 reported cstate change: config changed from index 3 to 4, VOTER 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129) evicted. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: false } } }
I20240501 13:56:35.024855 25998 ts_tablet_manager.cc:1888] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:35.025424 25998 log.cc:1196] T b085a215235d482e86b0a9529458b3c6 P ef020bad8d24499281f69de00e90af06: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-2-root/wals/b085a215235d482e86b0a9529458b3c6
I20240501 13:56:35.028132 24902 catalog_manager.cc:4830] TS ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317): tablet b085a215235d482e86b0a9529458b3c6 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:35.033092 25002 tablet_service.cc:1514] Processing DeleteTablet for tablet 56eceec85d2f42f5a0baf57bb03b45f4 with delete_type TABLET_DATA_TOMBSTONED (TS 346b2bb141744d459f1fa3ad602d1a15 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41354
I20240501 13:56:35.033989 25983 tablet_replica.cc:330] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: stopping tablet replica
I20240501 13:56:35.035041 25983 raft_consensus.cc:2229] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:35.035758 25983 raft_consensus.cc:2258] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:35.035816 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet bdc140167ab54483b3db40ba7a61937f with delete_type TABLET_DATA_TOMBSTONED (TS 171c5cc5f6df4820bc16946f8717737e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33652
I20240501 13:56:35.040573 25983 ts_tablet_manager.cc:1875] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:35.061652 25983 ts_tablet_manager.cc:1888] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:35.062114 25983 log.cc:1196] T 56eceec85d2f42f5a0baf57bb03b45f4 P 346b2bb141744d459f1fa3ad602d1a15: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-0-root/wals/56eceec85d2f42f5a0baf57bb03b45f4
I20240501 13:56:35.063560 24903 catalog_manager.cc:4830] TS 346b2bb141744d459f1fa3ad602d1a15 (127.23.70.129:33309): tablet 56eceec85d2f42f5a0baf57bb03b45f4 (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:35.065398 26004 tablet_replica.cc:330] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: stopping tablet replica
I20240501 13:56:35.066579 26004 raft_consensus.cc:2229] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:56:35.067343 26004 raft_consensus.cc:2258] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:56:35.070843 26004 ts_tablet_manager.cc:1875] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:56:35.082360 26004 ts_tablet_manager.cc:1888] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:56:35.082685 26004 log.cc:1196] T bdc140167ab54483b3db40ba7a61937f P 171c5cc5f6df4820bc16946f8717737e: Deleting WAL directory at /tmp/dist-test-taskBgks2G/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714571711146223-23834-0/minicluster-data/ts-1-root/wals/bdc140167ab54483b3db40ba7a61937f
I20240501 13:56:35.084149 24901 catalog_manager.cc:4830] TS 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553): tablet bdc140167ab54483b3db40ba7a61937f (table test-workload [id=bfde8915414949bb8bb4162b3ad7533b]) successfully deleted
I20240501 13:56:36.073848 25163 consensus_queue.cc:237] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:36.079098 25088 raft_consensus.cc:1261] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:36.080587 25132 consensus_peers.cc:485] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ba50ad350b4473cb475aa22f2ab76ef. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:36.081578 25013 raft_consensus.cc:1261] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.081274 26007 consensus_queue.cc:1035] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:36.083040 25996 consensus_queue.cc:1035] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:36.090330 25984 raft_consensus.cc:2932] T 7ba50ad350b4473cb475aa22f2ab76ef P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.092320 25086 raft_consensus.cc:2932] T 7ba50ad350b4473cb475aa22f2ab76ef P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.093106 25013 raft_consensus.cc:2932] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.113379 24913 catalog_manager.cc:5484] T 7ba50ad350b4473cb475aa22f2ab76ef P 346b2bb141744d459f1fa3ad602d1a15 reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.114804 25088 consensus_queue.cc:237] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
W20240501 13:56:36.124939 25058 consensus_peers.cc:485] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b8ee2e10bbd483e9356972bab5ea084. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:36.127249 25163 raft_consensus.cc:1261] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.139542 26005 consensus_queue.cc:1035] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:36.147517 25013 raft_consensus.cc:1261] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.152813 26005 consensus_queue.cc:1035] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:36.162101 26003 raft_consensus.cc:2932] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.163892 25163 raft_consensus.cc:2932] T 3b8ee2e10bbd483e9356972bab5ea084 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.179216 25086 consensus_queue.cc:237] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:36.183650 24914 catalog_manager.cc:5484] T 3b8ee2e10bbd483e9356972bab5ea084 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:36.184823 25013 raft_consensus.cc:2932] T 3b8ee2e10bbd483e9356972bab5ea084 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } attrs { replace: true } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
W20240501 13:56:36.206442 25058 consensus_peers.cc:485] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73740a9c3d7443cea7a555b0367a2b66. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:36.208288 25163 raft_consensus.cc:1261] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.209789 25013 raft_consensus.cc:1261] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer 171c5cc5f6df4820bc16946f8717737e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.213999 26002 consensus_queue.cc:1035] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:36.215564 26003 consensus_queue.cc:1035] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:56:36.230984 26020 raft_consensus.cc:2932] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.232528 25013 raft_consensus.cc:2932] T 73740a9c3d7443cea7a555b0367a2b66 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.248332 25161 raft_consensus.cc:2932] T 73740a9c3d7443cea7a555b0367a2b66 P ef020bad8d24499281f69de00e90af06 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.249365 25162 consensus_queue.cc:237] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } }
I20240501 13:56:36.259532 25086 raft_consensus.cc:1261] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:56:36.262326 24914 catalog_manager.cc:5484] T 73740a9c3d7443cea7a555b0367a2b66 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "171c5cc5f6df4820bc16946f8717737e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:56:36.262331 25984 consensus_queue.cc:1035] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240501 13:56:36.268682 25013 raft_consensus.cc:1261] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Refusing update from remote peer ef020bad8d24499281f69de00e90af06: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:56:36.269282 25132 consensus_peers.cc:485] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 -> Peer dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132:38635): Couldn't send request to peer dc721447aaca4b9ea5b345b35cf0e19e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 098d5f02b04442d2acdc15fcb9a7b429. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:56:36.270711 26010 consensus_queue.cc:1035] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 13:56:36.281229 25984 raft_consensus.cc:2932] T 098d5f02b04442d2acdc15fcb9a7b429 P ef020bad8d24499281f69de00e90af06 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.286360 25086 raft_consensus.cc:2932] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.288522 25013 raft_consensus.cc:2932] T 098d5f02b04442d2acdc15fcb9a7b429 P 346b2bb141744d459f1fa3ad602d1a15 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.314810 24915 catalog_manager.cc:5484] T 098d5f02b04442d2acdc15fcb9a7b429 P 171c5cc5f6df4820bc16946f8717737e reported cstate change: config changed from index -1 to 2, NON_VOTER dc721447aaca4b9ea5b345b35cf0e19e (127.23.70.132) added. New cstate: current_term: 1 leader_uuid: "ef020bad8d24499281f69de00e90af06" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346b2bb141744d459f1fa3ad602d1a15" member_type: VOTER last_known_addr { host: "127.23.70.129" port: 33309 } } peers { permanent_uuid: "171c5cc5f6df4820bc16946f8717737e" member_type: VOTER last_known_addr { host: "127.23.70.130" port: 34553 } attrs { replace: true } } peers { permanent_uuid: "ef020bad8d24499281f69de00e90af06" member_type: VOTER last_known_addr { host: "127.23.70.131" port: 40317 } } peers { permanent_uuid: "dc721447aaca4b9ea5b345b35cf0e19e" member_type: NON_VOTER last_known_addr { host: "127.23.70.132" port: 38635 } attrs { promote: true } } }
I20240501 13:56:36.557267 26026 ts_tablet_manager.cc:902] T 3b8ee2e10bbd483e9356972bab5ea084 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
F20240501 13:56:36.558118 26026 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0b683a5a430741e6ac1949303d905584.queue_time_us, tablet-download-pool-0b683a5a430741e6ac1949303d905584.queue_time_us) (tablet-download-pool-0b683a5a430741e6ac1949303d905584.run_cpu_time_us, tablet-download-pool-0b683a5a430741e6ac1949303d905584.run_cpu_time_us) (tablet-download-pool-0b683a5a430741e6ac1949303d905584.run_wall_time_us, tablet-download-pool-0b683a5a430741e6ac1949303d905584.run_wall_time_us) (tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.queue_time_us, tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.queue_time_us) (tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.run_cpu_time_us, tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.run_cpu_time_us) (tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.run_wall_time_us, tablet-download-pool-1b7e26b87b054270bc41f6e57ff69ac0.run_wall_time_us) (tablet-download-pool-3b8ee2e10bbd483e9356972bab5ea084.queue_time_us, tablet-download-pool-3b8ee2e10bbd483e9356972bab5ea084.queue_time_us) (tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.queue_time_us, tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.queue_time_us) (tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.run_cpu_time_us, tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.run_cpu_time_us) (tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.run_wall_time_us, tablet-download-pool-56eceec85d2f42f5a0baf57bb03b45f4.run_wall_time_us) (tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.queue_time_us, tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.queue_time_us) (tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.run_cpu_time_us, tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.run_cpu_time_us) (tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.run_wall_time_us, tablet-download-pool-7b159610f514496a8a1faf7d78c6c839.run_wall_time_us) (tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.queue_time_us, tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.queue_time_us) (tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.run_cpu_time_us, tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.run_cpu_time_us) (tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.run_wall_time_us, tablet-download-pool-872135f76cb04f6ebfce1899ed32a5a0.run_wall_time_us) (tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.queue_time_us, tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.queue_time_us) (tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.run_cpu_time_us, tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.run_cpu_time_us) (tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.run_wall_time_us, tablet-download-pool-a3e4ec10c6ef41a9815b71da267be33f.run_wall_time_us) (tablet-download-pool-b085a215235d482e86b0a9529458b3c6.queue_time_us, tablet-download-pool-b085a215235d482e86b0a9529458b3c6.queue_time_us) (tablet-download-pool-b085a215235d482e86b0a9529458b3c6.run_cpu_time_us, tablet-download-pool-b085a215235d482e86b0a9529458b3c6.run_cpu_time_us) (tablet-download-pool-b085a215235d482e86b0a9529458b3c6.run_wall_time_us, tablet-download-pool-b085a215235d482e86b0a9529458b3c6.run_wall_time_us) (tablet-download-pool-b14ca53436a04447b62816101d8b838b.queue_time_us, tablet-download-pool-b14ca53436a04447b62816101d8b838b.queue_time_us) (tablet-download-pool-b14ca53436a04447b62816101d8b838b.run_cpu_time_us, tablet-download-pool-b14ca53436a04447b62816101d8b838b.run_cpu_time_us) (tablet-download-pool-b14ca53436a04447b62816101d8b838b.run_wall_time_us, tablet-download-pool-b14ca53436a04447b62816101d8b838b.run_wall_time_us) (tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.queue_time_us, tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.queue_time_us) (tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.run_cpu_time_us, tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.run_cpu_time_us) (tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.run_wall_time_us, tablet-download-pool-bdc140167ab54483b3db40ba7a61937f.run_wall_time_us) (tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.queue_time_us, tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.queue_time_us) (tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.run_cpu_time_us, tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.run_cpu_time_us) (tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.run_wall_time_us, tablet-download-pool-bf58f127779f4ebd94273cd7fda138cd.run_wall_time_us) (tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.queue_time_us, tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.queue_time_us) (tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.run_cpu_time_us, tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.run_cpu_time_us) (tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.run_wall_time_us, tablet-download-pool-c77fc3f2c34f4b58bb2af4583495adba.run_wall_time_us) (tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.queue_time_us, tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.queue_time_us) (tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.run_cpu_time_us, tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.run_cpu_time_us) (tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.run_wall_time_us, tablet-download-pool-e059d8fd99574a5eac0b49e9405d7bd4.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714571796 (unix time) try "date -d @1714571796" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005d1a) received by PID 23834 (TID 0x7fc5319c5700) from PID 23834; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fc5ffe93980 (unknown) at ??:0
@ 0x7fc5f9407fb7 gsignal at ??:0
I20240501 13:56:36.647485 26029 ts_tablet_manager.cc:902] T 73740a9c3d7443cea7a555b0367a2b66 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer 171c5cc5f6df4820bc16946f8717737e (127.23.70.130:34553)
@ 0x7fc5f9409921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20240501 13:56:36.673671 26028 ts_tablet_manager.cc:902] T 7ba50ad350b4473cb475aa22f2ab76ef P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
@ 0x7fc5fc1fdda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fc5fc1fe950 google::LogMessage::Flush() at ??:0
@ 0x7fc5fc203a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5fcb8b3c7 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fc5fcb6194f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fc5fcb5f0a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fc60497f422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 13:56:36.849160 26042 ts_tablet_manager.cc:902] T 098d5f02b04442d2acdc15fcb9a7b429 P dc721447aaca4b9ea5b345b35cf0e19e: Initiating tablet copy from peer ef020bad8d24499281f69de00e90af06 (127.23.70.131:40317)
@ 0x7fc604985c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fc604a1b4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fc604a4c3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fc604a321ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fc604a3217a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7fc604a3210a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fc604a320d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fc604a312fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fc605dd7da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fc605dd7bda std::__1::function<>::operator()() at ??:0
@ 0x7fc5fcb63237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5fcb66692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fc5fcb6664a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7fc5fcb665da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7fc5fcb665a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fc5fcb6589e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7fc605dd7da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fc605dd7bda std::__1::function<>::operator()() at ??:0
@ 0x7fc5fcb45937 kudu::Thread::SuperviseThread() at ??:0