[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:38:29.397720 3527 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.113.254:34545
I20240502 21:38:29.399068 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:38:29.400131 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:38:29.418174 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:38:29.418272 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:29.419777 3527 server_base.cc:1034] running on GCE node
W20240502 21:38:29.418962 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:30.542229 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:38:30.542496 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:38:30.542713 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685910542690 us; error 0 us; skew 500 ppm
I20240502 21:38:30.543569 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:38:30.554482 3527 webserver.cc:460] Webserver started at http://127.3.113.254:39307/ using document root <none> and password file <none>
I20240502 21:38:30.555451 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:38:30.555692 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:38:30.556152 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:38:30.562757 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/master-0-root/instance:
uuid: "0b983aa8a13146f89eeadcef2301ce75"
format_stamp: "Formatted at 2024-05-02 21:38:30 on dist-test-slave-pddb"
I20240502 21:38:30.571041 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20240502 21:38:30.576571 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:30.577845 3527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 21:38:30.578236 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/master-0-root
uuid: "0b983aa8a13146f89eeadcef2301ce75"
format_stamp: "Formatted at 2024-05-02 21:38:30 on dist-test-slave-pddb"
I20240502 21:38:30.578629 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:38:30.645026 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:38:30.646642 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:38:30.647109 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:38:30.721556 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.254:34545
I20240502 21:38:30.721662 3594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.254:34545 every 8 connection(s)
I20240502 21:38:30.728885 3595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:30.753237 3595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: Bootstrap starting.
I20240502 21:38:30.758996 3595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:30.760906 3595 log.cc:824] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: Log is configured to *not* fsync() on all Append() calls
I20240502 21:38:30.765612 3595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: No bootstrap required, opened a new log
I20240502 21:38:30.788666 3595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER }
I20240502 21:38:30.789260 3595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:30.789474 3595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b983aa8a13146f89eeadcef2301ce75, State: Initialized, Role: FOLLOWER
I20240502 21:38:30.790231 3595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER }
I20240502 21:38:30.790726 3595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:38:30.790967 3595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:38:30.791258 3595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:30.795454 3595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER }
I20240502 21:38:30.796175 3595 leader_election.cc:302] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b983aa8a13146f89eeadcef2301ce75; no voters:
I20240502 21:38:30.797945 3595 leader_election.cc:288] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:38:30.798318 3598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:30.800630 3598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 1 LEADER]: Becoming Leader. State: Replica: 0b983aa8a13146f89eeadcef2301ce75, State: Running, Role: LEADER
I20240502 21:38:30.801631 3598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER }
I20240502 21:38:30.802753 3595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:38:30.813091 3600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b983aa8a13146f89eeadcef2301ce75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER } }
I20240502 21:38:30.813946 3600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [sys.catalog]: This master's current role is: LEADER
I20240502 21:38:30.813557 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b983aa8a13146f89eeadcef2301ce75. Latest consensus state: current_term: 1 leader_uuid: "0b983aa8a13146f89eeadcef2301ce75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b983aa8a13146f89eeadcef2301ce75" member_type: VOTER } }
I20240502 21:38:30.816145 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [sys.catalog]: This master's current role is: LEADER
I20240502 21:38:30.819365 3607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:38:30.830411 3607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:38:30.833352 3527 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:38:30.848299 3607 catalog_manager.cc:1347] Generated new cluster ID: 3c66e1df13e34ddc9d4cfe9b56ab496a
I20240502 21:38:30.848677 3607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:38:30.866405 3607 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:38:30.867978 3607 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:38:30.880419 3607 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: Generated new TSK 0
I20240502 21:38:30.881423 3607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:38:30.903905 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:38:30.912729 3617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:38:30.914477 3618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:38:30.918311 3620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:30.919121 3527 server_base.cc:1034] running on GCE node
I20240502 21:38:30.920151 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:38:30.920466 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:38:30.920692 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685910920668 us; error 0 us; skew 500 ppm
I20240502 21:38:30.921577 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:38:30.925977 3527 webserver.cc:460] Webserver started at http://127.3.113.193:45629/ using document root <none> and password file <none>
I20240502 21:38:30.926745 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:38:30.926999 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:38:30.927361 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:38:30.929013 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-0-root/instance:
uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
format_stamp: "Formatted at 2024-05-02 21:38:30 on dist-test-slave-pddb"
I20240502 21:38:30.936412 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240502 21:38:30.941989 3625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:30.942972 3527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20240502 21:38:30.943287 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
format_stamp: "Formatted at 2024-05-02 21:38:30 on dist-test-slave-pddb"
I20240502 21:38:30.943645 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:38:30.965713 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:38:30.967103 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:38:30.969569 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:38:30.973875 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:38:30.974149 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:30.974423 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:38:30.974594 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.070907 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.193:33159
I20240502 21:38:31.071049 3688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.193:33159 every 8 connection(s)
I20240502 21:38:31.080935 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:38:31.089996 3693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:31.105355 3689 heartbeater.cc:347] Connected to a master server at 127.3.113.254:34545
I20240502 21:38:31.106025 3689 heartbeater.cc:464] Registering TS with master...
I20240502 21:38:31.107645 3689 heartbeater.cc:511] Master 127.3.113.254:34545 requested a full tablet report, sending...
I20240502 21:38:31.112705 3560 ts_manager.cc:194] Registered new tserver with Master: c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:31.116883 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59752
W20240502 21:38:31.123961 3694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:31.127139 3527 server_base.cc:1034] running on GCE node
W20240502 21:38:31.127903 3696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:31.128793 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:38:31.128983 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:38:31.129169 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685911129148 us; error 0 us; skew 500 ppm
I20240502 21:38:31.129809 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:38:31.132736 3527 webserver.cc:460] Webserver started at http://127.3.113.194:36955/ using document root <none> and password file <none>
I20240502 21:38:31.133256 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:38:31.133462 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:38:31.133733 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:38:31.134865 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-1-root/instance:
uuid: "c0ab090735f1416987814c66f8cc9669"
format_stamp: "Formatted at 2024-05-02 21:38:31 on dist-test-slave-pddb"
I20240502 21:38:31.140381 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240502 21:38:31.144379 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.145289 3527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:38:31.145568 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
uuid: "c0ab090735f1416987814c66f8cc9669"
format_stamp: "Formatted at 2024-05-02 21:38:31 on dist-test-slave-pddb"
I20240502 21:38:31.145840 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:38:31.161610 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:38:31.162889 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:38:31.165426 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:38:31.168344 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:38:31.168557 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.168823 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:38:31.168985 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.267788 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.194:33571
I20240502 21:38:31.267921 3763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.194:33571 every 8 connection(s)
I20240502 21:38:31.274745 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:38:31.282835 3767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:38:31.286422 3768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:31.290169 3764 heartbeater.cc:347] Connected to a master server at 127.3.113.254:34545
I20240502 21:38:31.290606 3764 heartbeater.cc:464] Registering TS with master...
I20240502 21:38:31.297672 3764 heartbeater.cc:511] Master 127.3.113.254:34545 requested a full tablet report, sending...
W20240502 21:38:31.299384 3770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:38:31.300320 3560 ts_manager.cc:194] Registered new tserver with Master: c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:31.302098 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59768
I20240502 21:38:31.305480 3527 server_base.cc:1034] running on GCE node
I20240502 21:38:31.306526 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:38:31.306787 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:38:31.306989 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685911306971 us; error 0 us; skew 500 ppm
I20240502 21:38:31.307760 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:38:31.310743 3527 webserver.cc:460] Webserver started at http://127.3.113.195:40767/ using document root <none> and password file <none>
I20240502 21:38:31.311386 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:38:31.311630 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:38:31.311952 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:38:31.313539 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-2-root/instance:
uuid: "2f60f6bcac494ebf8c632930c129d75d"
format_stamp: "Formatted at 2024-05-02 21:38:31 on dist-test-slave-pddb"
I20240502 21:38:31.319507 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 21:38:31.323732 3775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.324599 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:38:31.324939 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
uuid: "2f60f6bcac494ebf8c632930c129d75d"
format_stamp: "Formatted at 2024-05-02 21:38:31 on dist-test-slave-pddb"
I20240502 21:38:31.325309 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714685909228215-3527-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:38:31.348241 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:38:31.349907 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:38:31.351610 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:38:31.355523 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:38:31.355806 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.356101 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:38:31.356341 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:38:31.445483 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.195:43271
I20240502 21:38:31.445541 3838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.195:43271 every 8 connection(s)
I20240502 21:38:31.462208 3839 heartbeater.cc:347] Connected to a master server at 127.3.113.254:34545
I20240502 21:38:31.462577 3839 heartbeater.cc:464] Registering TS with master...
I20240502 21:38:31.463281 3839 heartbeater.cc:511] Master 127.3.113.254:34545 requested a full tablet report, sending...
I20240502 21:38:31.465449 3560 ts_manager.cc:194] Registered new tserver with Master: 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:31.466396 3527 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012781647s
I20240502 21:38:31.467283 3560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59782
I20240502 21:38:31.470095 3527 test_util.cc:268] Using random seed: 1822920694
I20240502 21:38:31.521096 3560 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59794:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:38:31.525050 3560 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:38:31.629741 3724 tablet_service.cc:1467] Processing CreateTablet for tablet d61732cc5bb64686a0bad7d1c5a6d80d (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 21:38:31.634366 3728 tablet_service.cc:1467] Processing CreateTablet for tablet e113d6a5dcf34a4b9a3d585585215bda (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 21:38:31.645433 3724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61732cc5bb64686a0bad7d1c5a6d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.643688 3726 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8543cd0ff04c2598b1ea88638dd729 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 21:38:31.641126 3725 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99e3a145c849a1926accca79b2dbc3 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 21:38:31.640723 3727 tablet_service.cc:1467] Processing CreateTablet for tablet 925e68471ba24c42a502a8c6597f9e48 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 21:38:31.649808 3721 tablet_service.cc:1467] Processing CreateTablet for tablet e05b127281f0418dbc30f30bdfed01f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 21:38:31.649329 3728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e113d6a5dcf34a4b9a3d585585215bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.646373 3729 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7792908e264a6bb90e85b9042524f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 21:38:31.650415 3722 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddf42b3217464e9ea06cf6bc13df79 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 21:38:31.649416 3723 tablet_service.cc:1467] Processing CreateTablet for tablet c10aff62cc254785a05e04ba677a4d34 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 21:38:31.654245 3721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05b127281f0418dbc30f30bdfed01f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.657908 3726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8543cd0ff04c2598b1ea88638dd729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.661024 3729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7792908e264a6bb90e85b9042524f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.662312 3653 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7792908e264a6bb90e85b9042524f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 21:38:31.664551 3652 tablet_service.cc:1467] Processing CreateTablet for tablet e113d6a5dcf34a4b9a3d585585215bda (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 21:38:31.664239 3653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7792908e264a6bb90e85b9042524f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.666553 3723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10aff62cc254785a05e04ba677a4d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.670254 3652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e113d6a5dcf34a4b9a3d585585215bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.673074 3722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddf42b3217464e9ea06cf6bc13df79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.674849 3651 tablet_service.cc:1467] Processing CreateTablet for tablet 925e68471ba24c42a502a8c6597f9e48 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 21:38:31.672706 3650 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8543cd0ff04c2598b1ea88638dd729 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 21:38:31.673980 3649 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99e3a145c849a1926accca79b2dbc3 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 21:38:31.677954 3649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99e3a145c849a1926accca79b2dbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.678251 3725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99e3a145c849a1926accca79b2dbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.685735 3651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925e68471ba24c42a502a8c6597f9e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.686781 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925e68471ba24c42a502a8c6597f9e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.691306 3650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8543cd0ff04c2598b1ea88638dd729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.699524 3647 tablet_service.cc:1467] Processing CreateTablet for tablet c10aff62cc254785a05e04ba677a4d34 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 21:38:31.701948 3803 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7792908e264a6bb90e85b9042524f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240502 21:38:31.696496 3648 tablet_service.cc:1467] Processing CreateTablet for tablet d61732cc5bb64686a0bad7d1c5a6d80d (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 21:38:31.706480 3802 tablet_service.cc:1467] Processing CreateTablet for tablet e113d6a5dcf34a4b9a3d585585215bda (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240502 21:38:31.737074 3646 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddf42b3217464e9ea06cf6bc13df79 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 21:38:31.741636 3803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7792908e264a6bb90e85b9042524f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.742688 3647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10aff62cc254785a05e04ba677a4d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.743192 3801 tablet_service.cc:1467] Processing CreateTablet for tablet 925e68471ba24c42a502a8c6597f9e48 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240502 21:38:31.748004 3648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61732cc5bb64686a0bad7d1c5a6d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.751709 3800 tablet_service.cc:1467] Processing CreateTablet for tablet 6e8543cd0ff04c2598b1ea88638dd729 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240502 21:38:31.752625 3646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddf42b3217464e9ea06cf6bc13df79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.751973 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e113d6a5dcf34a4b9a3d585585215bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.753955 3645 tablet_service.cc:1467] Processing CreateTablet for tablet e05b127281f0418dbc30f30bdfed01f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 21:38:31.752295 3799 tablet_service.cc:1467] Processing CreateTablet for tablet 1a99e3a145c849a1926accca79b2dbc3 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240502 21:38:31.765524 3645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05b127281f0418dbc30f30bdfed01f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.783032 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a99e3a145c849a1926accca79b2dbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.793426 3800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e8543cd0ff04c2598b1ea88638dd729. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.797937 3801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 925e68471ba24c42a502a8c6597f9e48. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:38:31.852414 3608 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:38:31.825578 3798 tablet_service.cc:1467] Processing CreateTablet for tablet d61732cc5bb64686a0bad7d1c5a6d80d (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240502 21:38:31.859387 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61732cc5bb64686a0bad7d1c5a6d80d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.857262 3795 tablet_service.cc:1467] Processing CreateTablet for tablet e05b127281f0418dbc30f30bdfed01f2 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240502 21:38:31.837105 3797 tablet_service.cc:1467] Processing CreateTablet for tablet c10aff62cc254785a05e04ba677a4d34 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240502 21:38:31.861716 3796 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddf42b3217464e9ea06cf6bc13df79 (DEFAULT_TABLE table=test-workload [id=55897fa4bea1447b9d258fffac7e5c53]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240502 21:38:31.868695 3796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddf42b3217464e9ea06cf6bc13df79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.880841 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10aff62cc254785a05e04ba677a4d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:31.896624 3795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e05b127281f0418dbc30f30bdfed01f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:38:32.015523 3857 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.030866 3858 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.055193 3859 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.067014 3857 tablet_bootstrap.cc:654] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.067201 3858 tablet_bootstrap.cc:654] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.079176 3859 tablet_bootstrap.cc:654] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.087263 3857 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.087783 3857 ts_tablet_manager.cc:1367] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.008s
I20240502 21:38:32.090592 3857 raft_consensus.cc:348] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.091421 3857 raft_consensus.cc:374] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.091729 3857 raft_consensus.cc:729] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.092448 3857 consensus_queue.cc:260] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.097498 3859 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.098176 3859 ts_tablet_manager.cc:1367] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.013s
I20240502 21:38:32.100322 3857 ts_tablet_manager.cc:1398] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20240502 21:38:32.101173 3689 heartbeater.cc:503] Master 127.3.113.254:34545 was elected leader, sending a full tablet report...
I20240502 21:38:32.102032 3859 raft_consensus.cc:348] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.103089 3859 raft_consensus.cc:374] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.103539 3859 raft_consensus.cc:729] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.104724 3859 consensus_queue.cc:260] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.106263 3857 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.112843 3857 tablet_bootstrap.cc:654] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.113886 3839 heartbeater.cc:503] Master 127.3.113.254:34545 was elected leader, sending a full tablet report...
I20240502 21:38:32.114764 3859 ts_tablet_manager.cc:1398] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20240502 21:38:32.115738 3859 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.123988 3858 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.124523 3858 ts_tablet_manager.cc:1367] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.112s user 0.015s sys 0.015s
I20240502 21:38:32.124969 3859 tablet_bootstrap.cc:654] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.126060 3857 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.126605 3857 ts_tablet_manager.cc:1367] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.006s
I20240502 21:38:32.127701 3858 raft_consensus.cc:348] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.128669 3858 raft_consensus.cc:374] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.129705 3858 raft_consensus.cc:729] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.130415 3857 raft_consensus.cc:348] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.131623 3857 raft_consensus.cc:374] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.132087 3857 raft_consensus.cc:729] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.133062 3857 consensus_queue.cc:260] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.130534 3858 consensus_queue.cc:260] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.136076 3857 ts_tablet_manager.cc:1398] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:38:32.136602 3859 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.137059 3859 ts_tablet_manager.cc:1367] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s
I20240502 21:38:32.137192 3857 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.139905 3859 raft_consensus.cc:348] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.140851 3859 raft_consensus.cc:374] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.141158 3859 raft_consensus.cc:729] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.142060 3859 consensus_queue.cc:260] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.144186 3764 heartbeater.cc:503] Master 127.3.113.254:34545 was elected leader, sending a full tablet report...
I20240502 21:38:32.144608 3859 ts_tablet_manager.cc:1398] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 21:38:32.145620 3859 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.148456 3858 ts_tablet_manager.cc:1398] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20240502 21:38:32.150043 3858 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.151742 3857 tablet_bootstrap.cc:654] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.154670 3859 tablet_bootstrap.cc:654] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.169116 3858 tablet_bootstrap.cc:654] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.175673 3857 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.178149 3857 ts_tablet_manager.cc:1367] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.012s
I20240502 21:38:32.192346 3857 raft_consensus.cc:348] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.193174 3857 raft_consensus.cc:374] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.193493 3857 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.193989 3859 tablet_bootstrap.cc:492] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.194478 3859 ts_tablet_manager.cc:1367] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.010s
I20240502 21:38:32.194234 3857 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.196846 3857 ts_tablet_manager.cc:1398] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20240502 21:38:32.197801 3857 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.197453 3859 raft_consensus.cc:348] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.198204 3859 raft_consensus.cc:374] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.198572 3859 raft_consensus.cc:729] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.199267 3859 consensus_queue.cc:260] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.201679 3858 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.202154 3858 ts_tablet_manager.cc:1367] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.000s
W20240502 21:38:32.206251 3840 tablet.cc:2369] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:38:32.208547 3857 tablet_bootstrap.cc:654] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.208623 3859 ts_tablet_manager.cc:1398] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240502 21:38:32.209677 3859 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.209838 3858 raft_consensus.cc:348] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.210762 3858 raft_consensus.cc:374] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.211117 3858 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.211881 3858 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.216099 3858 ts_tablet_manager.cc:1398] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20240502 21:38:32.217207 3858 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.223469 3858 tablet_bootstrap.cc:654] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.226274 3867 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.226796 3867 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.229938 3867 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.233850 3857 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.234462 3857 ts_tablet_manager.cc:1367] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.016s
I20240502 21:38:32.238005 3857 raft_consensus.cc:348] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.238826 3857 raft_consensus.cc:374] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.239202 3857 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.240108 3857 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.241019 3864 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.241513 3864 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.244221 3864 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:32.248054 3857 ts_tablet_manager.cc:1398] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.013s user 0.003s sys 0.007s
I20240502 21:38:32.249289 3857 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.250254 3863 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.250869 3863 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.253795 3863 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.276456 3859 tablet_bootstrap.cc:654] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.298013 3858 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.300041 3864 raft_consensus.cc:482] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.304562 3864 raft_consensus.cc:504] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.305733 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.306223 3858 ts_tablet_manager.cc:1367] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.012s
I20240502 21:38:32.309000 3864 leader_election.cc:288] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:32.308939 3739 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:32.315661 3857 tablet_bootstrap.cc:654] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.318037 3626 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.319396 3863 raft_consensus.cc:2781] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.319766 3863 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.320086 3863 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.320580 3858 raft_consensus.cc:348] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.324575 3858 raft_consensus.cc:374] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.326901 3858 raft_consensus.cc:729] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.326951 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.330267 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.333180 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.331779 3661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.333524 3813 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.331693 3858 consensus_queue.cc:260] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
W20240502 21:38:32.336910 3776 leader_election.cc:341] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.339605 3858 ts_tablet_manager.cc:1398] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20240502 21:38:32.340746 3859 tablet_bootstrap.cc:492] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.341509 3859 ts_tablet_manager.cc:1367] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.132s user 0.014s sys 0.006s
I20240502 21:38:32.341645 3858 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.343027 3703 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:32.344424 3867 raft_consensus.cc:2781] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.345820 3867 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.347082 3867 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.357100 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.357789 3813 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:32.363914 3858 tablet_bootstrap.cc:654] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.365834 3867 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.353075 3863 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.368674 3663 raft_consensus.cc:2379] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ab090735f1416987814c66f8cc9669 in current term 1: Already voted for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in this term.
I20240502 21:38:32.371062 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.371177 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.373148 3739 raft_consensus.cc:2379] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f60f6bcac494ebf8c632930c129d75d in current term 1: Already voted for candidate c0ab090735f1416987814c66f8cc9669 in this term.
I20240502 21:38:32.372604 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.375267 3776 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d; no voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:32.375847 3867 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.375892 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.376381 3864 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.369009 3859 raft_consensus.cc:348] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.379014 3859 raft_consensus.cc:374] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:38:32.378736 3776 leader_election.cc:341] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.379534 3859 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.380355 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:32.381150 3776 leader_election.cc:302] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d; no voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:32.381729 3738 raft_consensus.cc:2379] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in current term 1: Already voted for candidate c0ab090735f1416987814c66f8cc9669 in this term.
I20240502 21:38:32.383380 3863 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.387130 3864 raft_consensus.cc:2726] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.392045 3813 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.392717 3864 raft_consensus.cc:2726] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.393987 3703 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:32.396416 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.396827 3867 raft_consensus.cc:2781] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.397322 3813 raft_consensus.cc:2379] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in current term 1: Already voted for candidate c0ab090735f1416987814c66f8cc9669 in this term.
I20240502 21:38:32.397749 3858 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.398348 3858 ts_tablet_manager.cc:1367] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.058s user 0.017s sys 0.001s
I20240502 21:38:32.394783 3859 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.398543 3627 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669
I20240502 21:38:32.400548 3863 raft_consensus.cc:2726] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.403985 3857 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.404537 3857 ts_tablet_manager.cc:1367] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.155s user 0.019s sys 0.005s
I20240502 21:38:32.400518 3859 ts_tablet_manager.cc:1398] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.058s user 0.008s sys 0.001s
I20240502 21:38:32.405536 3859 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.407336 3857 raft_consensus.cc:348] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.403860 3858 raft_consensus.cc:348] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.408159 3857 raft_consensus.cc:374] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.408807 3867 raft_consensus.cc:686] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:32.408830 3857 raft_consensus.cc:729] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.411362 3867 consensus_queue.cc:237] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.408638 3858 raft_consensus.cc:374] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.411814 3857 consensus_queue.cc:260] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.415182 3858 raft_consensus.cc:729] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.416236 3858 consensus_queue.cc:260] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.420449 3857 ts_tablet_manager.cc:1398] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 21:38:32.421231 3858 ts_tablet_manager.cc:1398] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20240502 21:38:32.424468 3857 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.427017 3864 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.427678 3864 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.432221 3858 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.438424 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.439123 3738 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 0.
I20240502 21:38:32.441249 3864 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:32.445022 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.445770 3663 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 0.
I20240502 21:38:32.448978 3857 tablet_bootstrap.cc:654] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.450968 3863 raft_consensus.cc:482] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.451728 3776 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:32.451629 3863 raft_consensus.cc:504] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.455427 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.458024 3863 leader_election.cc:288] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.458881 3859 tablet_bootstrap.cc:654] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
W20240502 21:38:32.460330 3626 leader_election.cc:341] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.462399 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
W20240502 21:38:32.468336 3627 leader_election.cc:341] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.468873 3627 leader_election.cc:302] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669
I20240502 21:38:32.469744 3863 raft_consensus.cc:2726] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.470327 3864 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.470834 3864 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.471076 3858 tablet_bootstrap.cc:654] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.471235 3864 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.479790 3864 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.481673 3859 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.482350 3859 ts_tablet_manager.cc:1367] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.011s
I20240502 21:38:32.483166 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:32.483870 3738 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.486214 3859 raft_consensus.cc:348] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.487247 3859 raft_consensus.cc:374] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.487630 3859 raft_consensus.cc:729] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.490008 3859 consensus_queue.cc:260] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.491209 3738 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 1.
I20240502 21:38:32.492480 3776 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:32.493690 3864 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:32.500030 3857 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.501156 3857 ts_tablet_manager.cc:1367] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.003s
I20240502 21:38:32.501259 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.501885 3663 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.503332 3559 catalog_manager.cc:5484] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.504079 3857 raft_consensus.cc:348] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.509377 3663 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 1.
I20240502 21:38:32.509588 3857 raft_consensus.cc:374] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.510058 3857 raft_consensus.cc:729] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.512375 3864 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.513262 3864 raft_consensus.cc:686] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:32.514338 3864 consensus_queue.cc:237] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:32.510885 3857 consensus_queue.cc:260] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.527626 3858 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.528360 3858 ts_tablet_manager.cc:1367] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.097s user 0.013s sys 0.006s
I20240502 21:38:32.533982 3859 ts_tablet_manager.cc:1398] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.051s user 0.010s sys 0.000s
I20240502 21:38:32.537747 3857 ts_tablet_manager.cc:1398] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20240502 21:38:32.546367 3857 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.539849 3859 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.532092 3858 raft_consensus.cc:348] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.556393 3858 raft_consensus.cc:374] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.556087 3864 raft_consensus.cc:482] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.556866 3864 raft_consensus.cc:504] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.558858 3857 tablet_bootstrap.cc:654] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.558480 3560 catalog_manager.cc:5484] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 1 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.560819 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.561614 3738 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 0.
I20240502 21:38:32.563920 3776 leader_election.cc:302] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:32.558897 3864 leader_election.cc:288] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:32.565068 3858 raft_consensus.cc:729] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.569223 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.568563 3888 raft_consensus.cc:2781] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.573575 3888 raft_consensus.cc:482] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.574261 3888 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:38:32.574441 3776 leader_election.cc:341] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.578003 3859 tablet_bootstrap.cc:654] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.582186 3888 raft_consensus.cc:504] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.582882 3858 consensus_queue.cc:260] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.585912 3858 ts_tablet_manager.cc:1398] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20240502 21:38:32.586768 3857 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.586961 3858 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.587354 3857 ts_tablet_manager.cc:1367] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.010s
I20240502 21:38:32.596069 3863 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.596168 3859 tablet_bootstrap.cc:492] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.596884 3859 ts_tablet_manager.cc:1367] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.008s
I20240502 21:38:32.596683 3863 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.598659 3863 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.599045 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.599745 3859 raft_consensus.cc:348] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.600523 3859 raft_consensus.cc:374] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.600375 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.600831 3859 raft_consensus.cc:729] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
W20240502 21:38:32.602156 3776 leader_election.cc:341] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.603358 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.604650 3858 tablet_bootstrap.cc:654] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.604557 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669"
W20240502 21:38:32.604526 3626 leader_election.cc:341] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.605305 3738 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.604589 3859 consensus_queue.cc:260] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
W20240502 21:38:32.610150 3627 leader_election.cc:341] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.610365 3859 ts_tablet_manager.cc:1398] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:38:32.610632 3627 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669
I20240502 21:38:32.611330 3859 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.611383 3863 raft_consensus.cc:2726] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.617789 3859 tablet_bootstrap.cc:654] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.618255 3888 leader_election.cc:288] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:32.617854 3738 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 1.
I20240502 21:38:32.619638 3867 raft_consensus.cc:482] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.620018 3867 raft_consensus.cc:504] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.620576 3776 leader_election.cc:302] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:32.621701 3867 leader_election.cc:288] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.622918 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.622934 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.623673 3663 raft_consensus.cc:2454] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.624217 3858 tablet_bootstrap.cc:492] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.624720 3858 ts_tablet_manager.cc:1367] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.000s
I20240502 21:38:32.625190 3702 leader_election.cc:302] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.626036 3867 raft_consensus.cc:2781] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.626571 3867 raft_consensus.cc:482] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 21:38:32.626605 3703 leader_election.cc:341] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.627151 3867 raft_consensus.cc:3037] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.628042 3888 raft_consensus.cc:2781] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.630235 3857 raft_consensus.cc:348] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.631657 3857 raft_consensus.cc:374] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.632047 3857 raft_consensus.cc:729] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.632938 3857 consensus_queue.cc:260] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.631233 3888 raft_consensus.cc:686] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:32.634873 3888 consensus_queue.cc:237] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.636183 3858 raft_consensus.cc:348] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.638620 3858 raft_consensus.cc:374] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.639277 3858 raft_consensus.cc:729] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.642724 3867 raft_consensus.cc:504] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.644765 3867 leader_election.cc:288] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.645162 3857 ts_tablet_manager.cc:1398] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.057s user 0.005s sys 0.003s
I20240502 21:38:32.645398 3858 consensus_queue.cc:260] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.646198 3857 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.647217 3858 ts_tablet_manager.cc:1398] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.022s user 0.004s sys 0.003s
I20240502 21:38:32.647907 3858 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.650297 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.651023 3663 raft_consensus.cc:3037] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.652782 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.654381 3857 tablet_bootstrap.cc:654] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
W20240502 21:38:32.655299 3703 leader_election.cc:341] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:38:32.658030 3858 tablet_bootstrap.cc:654] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.673547 3863 raft_consensus.cc:482] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.675411 3863 raft_consensus.cc:504] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.681711 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.681936 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.682449 3738 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:32.683089 3863 leader_election.cc:288] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.683909 3626 leader_election.cc:302] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.682492 3813 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:32.685425 3863 raft_consensus.cc:2781] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.685817 3863 raft_consensus.cc:482] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.686162 3863 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.689210 3859 tablet_bootstrap.cc:492] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.689438 3663 raft_consensus.cc:2454] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.689903 3859 ts_tablet_manager.cc:1367] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.007s
I20240502 21:38:32.690752 3702 leader_election.cc:302] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters: 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:32.691931 3867 raft_consensus.cc:2781] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.692471 3867 raft_consensus.cc:686] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:32.692832 3863 raft_consensus.cc:504] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.693315 3867 consensus_queue.cc:237] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.702384 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:32.694514 3859 raft_consensus.cc:348] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.703495 3738 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.706001 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.706634 3813 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.707202 3858 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.707695 3858 ts_tablet_manager.cc:1367] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.060s user 0.020s sys 0.022s
I20240502 21:38:32.703639 3859 raft_consensus.cc:374] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.708443 3859 raft_consensus.cc:729] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.711248 3858 raft_consensus.cc:348] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.712224 3858 raft_consensus.cc:374] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.712706 3858 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.712543 3859 consensus_queue.cc:260] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.713418 3858 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.716385 3859 ts_tablet_manager.cc:1398] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.026s user 0.011s sys 0.000s
I20240502 21:38:32.717226 3858 ts_tablet_manager.cc:1398] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240502 21:38:32.718194 3858 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.718647 3857 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.719107 3857 ts_tablet_manager.cc:1367] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.020s
I20240502 21:38:32.719425 3859 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.696655 3863 leader_election.cc:288] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.722662 3813 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 1.
I20240502 21:38:32.725239 3627 leader_election.cc:302] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.727329 3863 raft_consensus.cc:2781] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.727967 3858 tablet_bootstrap.cc:654] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.730118 3859 tablet_bootstrap.cc:654] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.730415 3560 catalog_manager.cc:5484] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.732484 3857 raft_consensus.cc:348] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.732734 3863 raft_consensus.cc:686] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:32.733247 3857 raft_consensus.cc:374] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.733565 3857 raft_consensus.cc:729] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.734184 3857 consensus_queue.cc:260] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.736438 3857 ts_tablet_manager.cc:1398] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.017s user 0.004s sys 0.003s
I20240502 21:38:32.737154 3858 tablet_bootstrap.cc:492] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.737293 3857 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Bootstrap starting.
I20240502 21:38:32.737658 3858 ts_tablet_manager.cc:1367] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20240502 21:38:32.723223 3738 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 1.
I20240502 21:38:32.740427 3858 raft_consensus.cc:348] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.741127 3858 raft_consensus.cc:374] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.738822 3559 catalog_manager.cc:5484] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 1 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } }
I20240502 21:38:32.741391 3858 raft_consensus.cc:729] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.734127 3863 consensus_queue.cc:237] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.745106 3857 tablet_bootstrap.cc:654] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.747617 3858 consensus_queue.cc:260] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.749977 3858 ts_tablet_manager.cc:1398] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 21:38:32.750846 3858 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: Bootstrap starting.
I20240502 21:38:32.756170 3858 tablet_bootstrap.cc:654] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.766268 3557 catalog_manager.cc:5484] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 1 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.770524 3857 tablet_bootstrap.cc:492] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: No bootstrap required, opened a new log
I20240502 21:38:32.771029 3857 ts_tablet_manager.cc:1367] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.004s
I20240502 21:38:32.773690 3857 raft_consensus.cc:348] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.774394 3857 raft_consensus.cc:374] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.774652 3857 raft_consensus.cc:729] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.775256 3857 consensus_queue.cc:260] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.778383 3857 ts_tablet_manager.cc:1398] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:38:32.783564 3858 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: No bootstrap required, opened a new log
I20240502 21:38:32.784066 3858 ts_tablet_manager.cc:1367] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.009s
I20240502 21:38:32.786608 3858 raft_consensus.cc:348] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.787371 3858 raft_consensus.cc:374] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.787648 3858 raft_consensus.cc:729] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.788292 3858 consensus_queue.cc:260] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.790589 3858 ts_tablet_manager.cc:1398] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:38:32.802057 3859 tablet_bootstrap.cc:492] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.802534 3859 ts_tablet_manager.cc:1367] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.084s user 0.031s sys 0.013s
I20240502 21:38:32.804848 3859 raft_consensus.cc:348] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.805727 3859 raft_consensus.cc:374] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.806025 3859 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.806653 3859 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.808738 3859 ts_tablet_manager.cc:1398] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 21:38:32.809597 3859 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: Bootstrap starting.
I20240502 21:38:32.815613 3859 tablet_bootstrap.cc:654] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: Neither blocks nor log segments found. Creating new log.
I20240502 21:38:32.820502 3859 tablet_bootstrap.cc:492] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: No bootstrap required, opened a new log
I20240502 21:38:32.820963 3859 ts_tablet_manager.cc:1367] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20240502 21:38:32.823323 3859 raft_consensus.cc:348] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.824035 3859 raft_consensus.cc:374] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:38:32.824371 3859 raft_consensus.cc:729] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Initialized, Role: FOLLOWER
I20240502 21:38:32.824997 3859 consensus_queue.cc:260] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.827468 3859 ts_tablet_manager.cc:1398] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:38:32.843797 3867 raft_consensus.cc:482] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.844435 3867 raft_consensus.cc:504] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.846513 3867 leader_election.cc:288] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.847568 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.847589 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.848325 3663 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.848439 3813 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.849280 3702 leader_election.cc:302] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.850227 3889 raft_consensus.cc:2781] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.850564 3889 raft_consensus.cc:482] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.850925 3889 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.854293 3867 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:38:32.856930 3608 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:38:32.857429 3889 raft_consensus.cc:504] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.860715 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.861338 3663 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.862874 3889 leader_election.cc:288] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.863896 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.864557 3812 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.867762 3663 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.868745 3702 leader_election.cc:302] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.869390 3889 raft_consensus.cc:2781] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.871277 3812 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.871284 3889 raft_consensus.cc:686] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:32.873296 3889 consensus_queue.cc:237] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.884878 3867 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:32.895406 3867 raft_consensus.cc:482] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.895983 3867 raft_consensus.cc:504] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.898967 3867 leader_election.cc:288] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.899859 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:32.900666 3663 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.904892 3702 leader_election.cc:302] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.906157 3867 raft_consensus.cc:2781] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:32.906674 3867 raft_consensus.cc:482] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:32.907133 3867 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.902961 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.908639 3813 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:32.919168 3560 catalog_manager.cc:5484] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.922683 3867 raft_consensus.cc:504] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.924685 3863 raft_consensus.cc:482] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:32.925223 3863 raft_consensus.cc:504] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.925485 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:32.926156 3662 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.927222 3863 leader_election.cc:288] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.928941 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:32.931622 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:32.932379 3738 raft_consensus.cc:2379] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in current term 1: Already voted for candidate 2f60f6bcac494ebf8c632930c129d75d in this term.
I20240502 21:38:32.932610 3662 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.933468 3626 leader_election.cc:302] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669
I20240502 21:38:32.933775 3702 leader_election.cc:302] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:32.934619 3863 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.935199 3889 raft_consensus.cc:2781] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:32.936391 3889 raft_consensus.cc:686] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:32.937436 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:32.935853 3867 leader_election.cc:288] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:32.938058 3813 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:32.937305 3889 consensus_queue.cc:237] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:32.942051 3863 raft_consensus.cc:2726] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:32.944026 3813 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:32.951257 3560 catalog_manager.cc:5484] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:32.963025 3888 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:32.974689 3924 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.038028 3889 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:33.038599 3889 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.040735 3889 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.041734 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:33.041697 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:33.042510 3813 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:33.042701 3662 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 0.
I20240502 21:38:33.043730 3703 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:33.044826 3889 raft_consensus.cc:2781] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:33.045416 3889 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:33.045696 3889 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.050431 3889 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.051970 3889 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.053114 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:33.053591 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:33.054241 3813 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.053776 3662 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.058925 3662 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:33.059957 3702 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.060698 3889 raft_consensus.cc:2781] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:33.060814 3813 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 1.
I20240502 21:38:33.061141 3889 raft_consensus.cc:686] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:33.062072 3889 consensus_queue.cc:237] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.070346 3559 catalog_manager.cc:5484] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:33.109583 3863 raft_consensus.cc:482] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:33.110168 3863 raft_consensus.cc:504] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.111763 3924 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.112893 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:33.113487 3863 leader_election.cc:288] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.113579 3738 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:33.114887 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d" is_pre_election: true
I20240502 21:38:33.114914 3626 leader_election.cc:302] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.115694 3813 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 0.
I20240502 21:38:33.116534 3907 raft_consensus.cc:2781] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:38:33.116902 3907 raft_consensus.cc:482] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:38:33.117219 3907 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.127044 3924 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.132170 3907 raft_consensus.cc:504] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.134649 3907 leader_election.cc:288] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.135006 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:33.135221 3889 consensus_queue.cc:1035] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.135653 3739 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.140147 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:33.140831 3813 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.145897 3739 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 1.
I20240502 21:38:33.147711 3626 leader_election.cc:302] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.149222 3863 raft_consensus.cc:2781] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:38:33.149886 3863 raft_consensus.cc:686] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:33.149489 3813 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 1.
I20240502 21:38:33.151082 3863 consensus_queue.cc:237] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.162941 3924 raft_consensus.cc:482] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:38:33.163470 3924 raft_consensus.cc:504] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.166532 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" is_pre_election: true
I20240502 21:38:33.167821 3924 leader_election.cc:288] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:33.169412 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a99e3a145c849a1926accca79b2dbc3" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ab090735f1416987814c66f8cc9669" is_pre_election: true
I20240502 21:38:33.174706 3813 raft_consensus.cc:3037] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:38:33.180701 3776 leader_election.cc:302] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d; no voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:33.182689 3560 catalog_manager.cc:5484] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 0 to 1, leader changed from <none> to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 1 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:33.187553 3924 raft_consensus.cc:2726] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:38:33.188930 3916 consensus_queue.cc:1035] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.330699 3916 consensus_queue.cc:1035] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.342991 3916 consensus_queue.cc:1035] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.345321 3863 consensus_queue.cc:1035] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.363404 3907 consensus_queue.cc:1035] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.476857 3813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:37658
I20240502 21:38:33.477625 3813 raft_consensus.cc:595] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:33.491916 3916 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.492777 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:60186
I20240502 21:38:33.493403 3738 raft_consensus.cc:595] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:33.506395 3867 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.513006 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:60186
I20240502 21:38:33.513684 3738 raft_consensus.cc:595] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:33.522785 3867 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.531744 3813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:37658
I20240502 21:38:33.532539 3813 raft_consensus.cc:595] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:33.541319 3916 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.546039 3924 raft_consensus.cc:979] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:33.546526 3924 raft_consensus.cc:1067] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:33.548442 3662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59562
I20240502 21:38:33.549101 3662 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:33.549489 3662 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.556463 3662 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:33.559062 3662 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.559345 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:33.561671 3738 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.568737 3738 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:33.559983 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:33.569676 3813 raft_consensus.cc:3032] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:33.570032 3813 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:33.570125 3626 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.571249 3936 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:33.570962 3813 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:33.571803 3936 raft_consensus.cc:686] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:33.572582 3813 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.572714 3936 consensus_queue.cc:237] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:33.579378 3813 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:33.581787 3559 catalog_manager.cc:5484] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 1 to 2, leader changed from 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 2 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } }
I20240502 21:38:33.590013 3662 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
mode: GRACEFUL
new_leader_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:59566
I20240502 21:38:33.590673 3662 raft_consensus.cc:595] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Received request to transfer leadership to TS 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:33.647658 3936 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:38:33.665793 3863 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:38:33.704908 3888 raft_consensus.cc:979] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:33.705487 3864 raft_consensus.cc:1067] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:33.707080 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60182
I20240502 21:38:33.707720 3738 raft_consensus.cc:482] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:33.708076 3738 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.714507 3738 raft_consensus.cc:504] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.716471 3738 leader_election.cc:288] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 2 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.717538 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:33.717574 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:33.718242 3662 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.718327 3812 raft_consensus.cc:3032] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:33.718705 3812 raft_consensus.cc:729] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:33.719416 3812 consensus_queue.cc:260] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.720659 3812 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.724491 3662 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 2.
I20240502 21:38:33.725579 3702 leader_election.cc:302] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.726372 3915 raft_consensus.cc:2781] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:33.726783 3915 raft_consensus.cc:686] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:33.727433 3812 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 2.
I20240502 21:38:33.727691 3915 consensus_queue.cc:237] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.734556 3560 catalog_manager.cc:5484] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 1 to 2, leader changed from 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:33.830840 3907 raft_consensus.cc:979] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: : Instructing follower 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:33.831363 3863 raft_consensus.cc:1067] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Signalling peer 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:33.832935 3812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:37642
I20240502 21:38:33.833459 3812 raft_consensus.cc:482] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:33.833770 3812 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.839460 3812 raft_consensus.cc:504] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.841658 3812 leader_election.cc:288] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 2 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:33.843423 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:33.843215 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e05b127281f0418dbc30f30bdfed01f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:33.844184 3738 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.844273 3662 raft_consensus.cc:3032] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:33.844902 3662 raft_consensus.cc:729] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:33.846124 3662 consensus_queue.cc:260] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.847568 3662 raft_consensus.cc:3037] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.852566 3738 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 2.
I20240502 21:38:33.853930 3776 leader_election.cc:302] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:33.854804 3864 raft_consensus.cc:2781] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:33.854951 3662 raft_consensus.cc:2454] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 2.
I20240502 21:38:33.855333 3864 raft_consensus.cc:686] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:33.856230 3864 consensus_queue.cc:237] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.864982 3560 catalog_manager.cc:5484] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 1 to 2, leader changed from c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193) to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 2 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } }
I20240502 21:38:33.916393 3916 raft_consensus.cc:979] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:33.916841 3915 raft_consensus.cc:1067] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:33.918352 3662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59556
I20240502 21:38:33.918825 3662 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:33.919109 3662 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.923497 3662 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.925103 3662 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:33.926146 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:33.926143 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:33.926808 3738 raft_consensus.cc:3032] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:33.926934 3812 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.927076 3738 raft_consensus.cc:729] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:33.927809 3738 consensus_queue.cc:260] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.928794 3738 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:33.931810 3812 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:33.933034 3627 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:33.933564 3738 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:33.933955 3863 raft_consensus.cc:2781] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:33.934406 3863 raft_consensus.cc:686] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:33.935184 3863 consensus_queue.cc:237] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:33.941869 3560 catalog_manager.cc:5484] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 1 to 2, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 2 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:34.009179 3812 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.010349 3863 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.018983 3738 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.021464 3863 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:38:34.078253 3916 raft_consensus.cc:979] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:34.078678 3915 raft_consensus.cc:1067] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:34.080047 3662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59556
I20240502 21:38:34.080552 3662 raft_consensus.cc:482] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:34.080840 3662 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:34.086088 3662 raft_consensus.cc:504] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:34.087587 3662 leader_election.cc:288] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:34.088605 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:34.088518 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:34.089269 3738 raft_consensus.cc:3032] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:34.089239 3812 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:34.089852 3738 raft_consensus.cc:729] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:34.090680 3738 consensus_queue.cc:260] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:34.091969 3738 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:34.096411 3812 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:34.097545 3627 leader_election.cc:302] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:34.098233 3863 raft_consensus.cc:2781] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:34.098258 3738 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:34.098850 3863 raft_consensus.cc:686] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:34.099572 3863 consensus_queue.cc:237] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:34.105305 3560 catalog_manager.cc:5484] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 1 to 2, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 2 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:34.133560 3812 raft_consensus.cc:1261] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.134773 3915 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.142762 3662 raft_consensus.cc:1261] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.144786 3889 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.325891 3738 raft_consensus.cc:1261] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.327292 3863 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.337668 3812 raft_consensus.cc:1261] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.339972 3949 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:38:34.342402 3738 raft_consensus.cc:1261] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.344450 3864 consensus_queue.cc:1035] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:38:34.358501 3662 raft_consensus.cc:1261] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.360725 3864 consensus_queue.cc:1035] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.563174 3812 raft_consensus.cc:1261] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.564465 3949 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:34.574524 3738 raft_consensus.cc:1261] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:34.575910 3863 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:36.614380 3527 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240502 21:38:36.615142 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:36.638211 3662 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
mode: GRACEFUL
new_leader_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:59580
I20240502 21:38:36.638720 3662 raft_consensus.cc:595] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Received request to transfer leadership to TS 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:36.648855 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:36.649423 3738 raft_consensus.cc:595] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Received request to transfer leadership to TS 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:36.650569 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:38:36.651005 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:36.654614 3863 raft_consensus.cc:979] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: : Instructing follower 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:36.655020 3962 raft_consensus.cc:1067] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Signalling peer 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:36.656476 3813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:37642
I20240502 21:38:36.656947 3813 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:36.657238 3813 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:36.662662 3813 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:36.664477 3813 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 3 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:36.665390 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:36.665642 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:36.665930 3738 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:36.666143 3662 raft_consensus.cc:3032] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:36.666441 3662 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:36.667177 3662 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:36.668112 3662 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:36.670715 3738 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 3.
I20240502 21:38:36.671890 3776 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:36.672654 3957 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:36.673149 3662 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 3.
I20240502 21:38:36.674553 3957 raft_consensus.cc:686] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:36.675360 3957 consensus_queue.cc:237] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:36.683591 3560 catalog_manager.cc:5484] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 2 to 3, leader changed from c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193) to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 3 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:37.060935 3738 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:37.062407 3957 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:37.075012 3662 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:37.076509 3971 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:38:37.119815 3889 raft_consensus.cc:979] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: : Instructing follower 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:37.120250 3975 raft_consensus.cc:1067] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Signalling peer 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:37.121675 3812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:37638
I20240502 21:38:37.122234 3812 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:37.122543 3812 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:37.127115 3812 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:37.128625 3812 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 2 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:37.129570 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:37.130075 3663 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:37.129928 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:37.130590 3738 raft_consensus.cc:3032] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:37.130821 3738 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:37.131425 3738 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:37.132428 3738 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:37.134865 3663 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 2.
I20240502 21:38:37.135869 3776 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:37.136762 3738 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 2.
I20240502 21:38:37.136866 3971 raft_consensus.cc:2781] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:37.137439 3971 raft_consensus.cc:686] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:37.138531 3971 consensus_queue.cc:237] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:37.145351 3560 catalog_manager.cc:5484] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 1 to 2, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 2 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } }
I20240502 21:38:37.652053 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:37.658114 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:37.658550 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:37.680089 3738 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:37.681370 3957 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:37.690286 3663 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:37.692250 3963 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:38.659384 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:38.664575 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:38.665066 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:39.666692 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:39.674672 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:39.675072 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:40.675908 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:40.680660 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:40.681073 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:41.681877 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:41.687805 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:41.688198 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:42.689085 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:42.695221 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:42.695771 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:43.696771 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:43.701516 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:43.701898 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:44.702842 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:44.708050 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:44.708496 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:45.709218 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:45.713874 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:45.714232 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:46.738652 3527 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240502 21:38:46.743075 3813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:37658
I20240502 21:38:46.743783 3813 raft_consensus.cc:595] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:46.748569 3663 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:59566
I20240502 21:38:46.749006 3663 raft_consensus.cc:595] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:46.753224 3663 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:59566
I20240502 21:38:46.753667 3663 raft_consensus.cc:595] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:46.757725 3663 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:59566
I20240502 21:38:46.758124 3663 raft_consensus.cc:595] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:46.764405 3813 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
mode: GRACEFUL
new_leader_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:37658
I20240502 21:38:46.764869 3813 raft_consensus.cc:595] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Received request to transfer leadership to TS c0ab090735f1416987814c66f8cc9669
I20240502 21:38:46.988543 3986 raft_consensus.cc:979] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:46.989107 3988 raft_consensus.cc:1067] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:46.990620 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60182
I20240502 21:38:46.991026 3738 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:46.991317 3738 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:46.996229 3738 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:46.998035 3738 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 4 election: Requested vote from peers 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271), c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:46.999150 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:46.999078 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:46.999974 3663 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:47.000069 3813 raft_consensus.cc:3032] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:38:47.000559 3813 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:47.001308 3813 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:47.002480 3813 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:47.005824 3663 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 4.
I20240502 21:38:47.006953 3813 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 4.
I20240502 21:38:47.006989 3702 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:47.007969 3975 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:38:47.008996 3975 raft_consensus.cc:686] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:47.010545 3975 consensus_queue.cc:237] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:47.013283 3983 raft_consensus.cc:979] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.013924 3990 raft_consensus.cc:1067] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.015712 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60170
I20240502 21:38:47.016112 3738 raft_consensus.cc:482] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:47.016412 3738 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:47.018935 3559 catalog_manager.cc:5484] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 3 to 4, leader changed from 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 4 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:47.023419 3738 raft_consensus.cc:504] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.025094 3738 leader_election.cc:288] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 2 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:47.026384 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:47.026997 3813 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:47.027352 3986 raft_consensus.cc:979] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.027812 3988 raft_consensus.cc:1067] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.029071 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60182
I20240502 21:38:47.029445 3738 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:47.029655 3738 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.033295 3663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:47.033929 3663 raft_consensus.cc:3032] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:47.034251 3663 raft_consensus.cc:729] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:47.034600 3738 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.035012 3663 consensus_queue.cc:260] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.035969 3813 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 2.
I20240502 21:38:47.036332 3663 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:47.037396 3738 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:47.037410 3703 leader_election.cc:302] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:47.038893 3975 raft_consensus.cc:2781] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:47.039355 3975 raft_consensus.cc:686] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:47.040226 3975 consensus_queue.cc:237] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.041783 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:47.042618 3662 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.042927 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:47.043478 3812 raft_consensus.cc:3032] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:47.043473 3663 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 2.
I20240502 21:38:47.043923 3812 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:47.044848 3812 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.046067 3812 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.051959 3662 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.053457 3702 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:47.054845 3989 raft_consensus.cc:2781] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:47.055033 3812 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.055415 3989 raft_consensus.cc:686] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:47.056852 3989 consensus_queue.cc:237] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.065364 3559 catalog_manager.cc:5484] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 1 to 2, leader changed from c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:47.066460 3559 catalog_manager.cc:5484] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 2 to 3, leader changed from 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 3 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:47.156764 3983 raft_consensus.cc:979] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.157223 3990 raft_consensus.cc:1067] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.158553 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60170
I20240502 21:38:47.158980 3738 raft_consensus.cc:482] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:47.159265 3738 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.163187 3738 raft_consensus.cc:504] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.164539 3738 leader_election.cc:288] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:47.165377 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:47.165520 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e8543cd0ff04c2598b1ea88638dd729" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:47.165874 3662 raft_consensus.cc:3032] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:47.165980 3812 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.166177 3662 raft_consensus.cc:729] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:47.166898 3662 consensus_queue.cc:260] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.167971 3662 raft_consensus.cc:3037] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.171072 3812 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.172025 3703 leader_election.cc:302] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:47.172498 3662 raft_consensus.cc:2454] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.172904 3989 raft_consensus.cc:2781] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:47.173333 3989 raft_consensus.cc:686] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:47.174062 3989 consensus_queue.cc:237] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.180941 3559 catalog_manager.cc:5484] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 2 to 3, leader changed from c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 3 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:47.202275 3983 raft_consensus.cc:979] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: : Instructing follower c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.202687 3990 raft_consensus.cc:1067] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Signalling peer c0ab090735f1416987814c66f8cc9669 to start an election
I20240502 21:38:47.204165 3738 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
from {username='slave'} at 127.0.0.1:60170
I20240502 21:38:47.204618 3738 raft_consensus.cc:482] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:47.204877 3738 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.209056 3738 raft_consensus.cc:504] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.210316 3738 leader_election.cc:288] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:47.211251 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:47.211342 3812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e113d6a5dcf34a4b9a3d585585215bda" candidate_uuid: "c0ab090735f1416987814c66f8cc9669" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:47.211853 3662 raft_consensus.cc:3032] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:47.211917 3812 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.212100 3662 raft_consensus.cc:729] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:47.212781 3662 consensus_queue.cc:260] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.213779 3662 raft_consensus.cc:3037] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:47.217548 3812 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.218497 3703 leader_election.cc:302] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c0ab090735f1416987814c66f8cc9669; no voters:
I20240502 21:38:47.218796 3662 raft_consensus.cc:2454] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ab090735f1416987814c66f8cc9669 in term 3.
I20240502 21:38:47.219354 3989 raft_consensus.cc:2781] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:47.219835 3989 raft_consensus.cc:686] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Becoming Leader. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:47.220517 3989 consensus_queue.cc:237] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:47.227305 3559 catalog_manager.cc:5484] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 reported cstate change: term changed from 2 to 3, leader changed from c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193) to c0ab090735f1416987814c66f8cc9669 (127.3.113.194). New cstate: current_term: 3 leader_uuid: "c0ab090735f1416987814c66f8cc9669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:47.546732 3662 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:38:47.548051 3989 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:38:47.560086 3663 raft_consensus.cc:1261] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:47.561821 3989 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:47.562498 3812 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:38:47.564071 3989 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:38:47.568776 3663 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.571056 3975 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:38:47.574152 3813 raft_consensus.cc:1261] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:47.575423 3975 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:47.597968 3812 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.599345 3989 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:47.605849 3662 raft_consensus.cc:1261] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.608897 3989 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:47.618726 3812 raft_consensus.cc:1261] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.620430 3975 consensus_queue.cc:1035] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:47.672438 3662 raft_consensus.cc:1261] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.673610 3975 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:47.682005 3812 raft_consensus.cc:1261] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Refusing update from remote peer c0ab090735f1416987814c66f8cc9669: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:47.684098 4011 consensus_queue.cc:1035] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:49.799188 3527 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240502 21:38:49.800004 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:49.807130 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:49.807889 3738 raft_consensus.cc:595] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 LEADER]: Received request to transfer leadership to TS 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:49.810549 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:49.811038 3738 raft_consensus.cc:595] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:49.813107 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:49.813560 3738 raft_consensus.cc:595] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:49.815573 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:49.816105 3738 raft_consensus.cc:595] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Received request to transfer leadership to TS 2f60f6bcac494ebf8c632930c129d75d
I20240502 21:38:49.818097 3738 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "c0ab090735f1416987814c66f8cc9669"
mode: GRACEFUL
new_leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:34448
I20240502 21:38:49.818533 3738 raft_consensus.cc:595] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Received request to transfer leadership to TS c5a1ad9a56f94dd3a5cb3e92c944ff1d
I20240502 21:38:49.819329 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240502 21:38:49.819769 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:49.861297 4002 raft_consensus.cc:979] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: : Instructing follower 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:49.861852 4017 raft_consensus.cc:1067] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 LEADER]: Signalling peer 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:49.863353 3813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:37638
I20240502 21:38:49.863863 3813 raft_consensus.cc:482] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:49.864182 3813 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:38:49.872442 3813 raft_consensus.cc:504] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:49.874769 3813 leader_election.cc:288] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 5 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159)
I20240502 21:38:49.875798 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:49.875902 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7792908e264a6bb90e85b9042524f2" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:49.876711 3738 raft_consensus.cc:3032] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:38:49.876843 3662 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:38:49.877084 3738 raft_consensus.cc:729] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:49.877933 3738 consensus_queue.cc:260] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:49.879354 3738 raft_consensus.cc:3037] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:38:49.884092 3662 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 5.
I20240502 21:38:49.885586 3776 leader_election.cc:302] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:49.886420 3738 raft_consensus.cc:2454] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 5.
I20240502 21:38:49.886605 4015 raft_consensus.cc:2781] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:38:49.888005 4015 raft_consensus.cc:686] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 5 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:49.888973 4015 consensus_queue.cc:237] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } }
I20240502 21:38:49.896574 3560 catalog_manager.cc:5484] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 4 to 5, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 5 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:50.065090 4002 raft_consensus.cc:979] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: : Instructing follower 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:50.065572 4017 raft_consensus.cc:1067] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Signalling peer 2f60f6bcac494ebf8c632930c129d75d to start an election
I20240502 21:38:50.067175 3813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34"
dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
from {username='slave'} at 127.0.0.1:37638
I20240502 21:38:50.067636 3813 raft_consensus.cc:482] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:50.067868 3813 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.073359 3813 raft_consensus.cc:504] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.074869 3813 leader_election.cc:288] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 3 election: Requested vote from peers c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159), c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571)
I20240502 21:38:50.075712 3662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
I20240502 21:38:50.076289 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10aff62cc254785a05e04ba677a4d34" candidate_uuid: "2f60f6bcac494ebf8c632930c129d75d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:50.076490 3662 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.076925 3738 raft_consensus.cc:3032] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:50.077278 3738 raft_consensus.cc:729] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:50.077981 3738 consensus_queue.cc:260] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.079077 3738 raft_consensus.cc:3037] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.079730 4002 raft_consensus.cc:979] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.080240 4017 raft_consensus.cc:1067] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.082024 3663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59556
I20240502 21:38:50.082578 3663 raft_consensus.cc:482] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:50.082960 3663 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.083011 3662 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 3.
I20240502 21:38:50.084069 3738 raft_consensus.cc:2454] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f60f6bcac494ebf8c632930c129d75d in term 3.
I20240502 21:38:50.084120 3776 leader_election.cc:302] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:50.085047 4015 raft_consensus.cc:2781] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:50.085455 4015 raft_consensus.cc:686] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Becoming Leader. State: Replica: 2f60f6bcac494ebf8c632930c129d75d, State: Running, Role: LEADER
I20240502 21:38:50.086297 4015 consensus_queue.cc:237] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.088620 3663 raft_consensus.cc:504] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.091292 3663 leader_election.cc:288] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 3 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:50.092156 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:50.092742 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "925e68471ba24c42a502a8c6597f9e48" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:50.092792 3738 raft_consensus.cc:3032] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:38:50.093251 3738 raft_consensus.cc:729] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:50.093814 3738 consensus_queue.cc:260] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.094024 3560 catalog_manager.cc:5484] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d reported cstate change: term changed from 2 to 3, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195). New cstate: current_term: 3 leader_uuid: "2f60f6bcac494ebf8c632930c129d75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: HEALTHY } } }
I20240502 21:38:50.094568 3738 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.094880 3813 raft_consensus.cc:3037] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:38:50.099843 3738 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 3.
I20240502 21:38:50.100028 3813 raft_consensus.cc:2454] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 3.
I20240502 21:38:50.100982 3626 leader_election.cc:302] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ab090735f1416987814c66f8cc9669, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:50.103135 4025 raft_consensus.cc:2781] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:38:50.104617 4025 raft_consensus.cc:686] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:50.105454 4025 consensus_queue.cc:237] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.112558 3560 catalog_manager.cc:5484] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 2 to 3, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 3 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:50.183560 4002 raft_consensus.cc:979] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.184026 4017 raft_consensus.cc:1067] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.185389 3663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59556
I20240502 21:38:50.185825 3663 raft_consensus.cc:482] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:50.186048 3663 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:50.190179 3663 raft_consensus.cc:504] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.191498 3663 leader_election.cc:288] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:50.192468 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:50.192965 3738 raft_consensus.cc:3032] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:38:50.192881 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cddf42b3217464e9ea06cf6bc13df79" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:50.193302 3738 raft_consensus.cc:729] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:50.193544 3813 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:50.194072 3738 consensus_queue.cc:260] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.195268 3738 raft_consensus.cc:3037] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:38:50.198961 3813 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:50.199543 3738 raft_consensus.cc:2454] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 2.
I20240502 21:38:50.200035 3627 leader_election.cc:302] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:50.200860 4025 raft_consensus.cc:2781] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:38:50.201243 4025 raft_consensus.cc:686] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:50.201920 4025 consensus_queue.cc:237] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.208830 3560 catalog_manager.cc:5484] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 1 to 2, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 2 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:50.270606 4002 raft_consensus.cc:979] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: : Instructing follower c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.271139 4017 raft_consensus.cc:1067] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Signalling peer c5a1ad9a56f94dd3a5cb3e92c944ff1d to start an election
I20240502 21:38:50.272703 3663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d"
dest_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d"
from {username='slave'} at 127.0.0.1:59556
I20240502 21:38:50.273126 3663 raft_consensus.cc:482] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:38:50.273348 3663 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:50.277822 3663 raft_consensus.cc:504] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.279213 3663 leader_election.cc:288] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 4 election: Requested vote from peers c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571), 2f60f6bcac494ebf8c632930c129d75d (127.3.113.195:43271)
I20240502 21:38:50.280040 3738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c0ab090735f1416987814c66f8cc9669"
I20240502 21:38:50.280097 3813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61732cc5bb64686a0bad7d1c5a6d80d" candidate_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2f60f6bcac494ebf8c632930c129d75d"
I20240502 21:38:50.280773 3738 raft_consensus.cc:3032] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:38:50.280851 3813 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:50.281150 3738 raft_consensus.cc:729] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c0ab090735f1416987814c66f8cc9669, State: Running, Role: LEADER
I20240502 21:38:50.281894 3738 consensus_queue.cc:260] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.283085 3738 raft_consensus.cc:3037] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:38:50.287093 3813 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 4.
I20240502 21:38:50.288173 3627 leader_election.cc:302] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f60f6bcac494ebf8c632930c129d75d, c5a1ad9a56f94dd3a5cb3e92c944ff1d; no voters:
I20240502 21:38:50.288833 3738 raft_consensus.cc:2454] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5a1ad9a56f94dd3a5cb3e92c944ff1d in term 4.
I20240502 21:38:50.288964 4025 raft_consensus.cc:2781] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:38:50.289481 4025 raft_consensus.cc:686] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 LEADER]: Becoming Leader. State: Replica: c5a1ad9a56f94dd3a5cb3e92c944ff1d, State: Running, Role: LEADER
I20240502 21:38:50.290298 4025 consensus_queue.cc:237] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } }
I20240502 21:38:50.297585 3560 catalog_manager.cc:5484] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d reported cstate change: term changed from 3 to 4, leader changed from c0ab090735f1416987814c66f8cc9669 (127.3.113.194) to c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193). New cstate: current_term: 4 leader_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 21:38:50.362722 3663 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 5 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:38:50.363881 4015 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:38:50.377990 3738 raft_consensus.cc:1261] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 5 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:38:50.379262 4015 consensus_queue.cc:1035] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:38:50.488783 3663 raft_consensus.cc:1261] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:50.489876 4015 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a1ad9a56f94dd3a5cb3e92c944ff1d" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 33159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:50.498991 3738 raft_consensus.cc:1261] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:50.500463 4025 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:38:50.501471 3739 raft_consensus.cc:1261] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Refusing update from remote peer 2f60f6bcac494ebf8c632930c129d75d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:50.504396 4033 consensus_queue.cc:1035] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 21:38:50.515251 3813 raft_consensus.cc:1261] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:38:50.517006 4026 consensus_queue.cc:1035] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:38:50.614631 3813 raft_consensus.cc:1261] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:50.615891 4026 consensus_queue.cc:1035] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:50.624720 3739 raft_consensus.cc:1261] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:38:50.626945 4026 consensus_queue.cc:1035] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:38:50.700379 3813 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:38:50.701661 4026 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f60f6bcac494ebf8c632930c129d75d" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:38:50.710460 3739 raft_consensus.cc:1261] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Refusing update from remote peer c5a1ad9a56f94dd3a5cb3e92c944ff1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:38:50.712955 4025 consensus_queue.cc:1035] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ab090735f1416987814c66f8cc9669" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:38:50.820709 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:50.825659 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:50.826074 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:51.826809 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:51.831606 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:51.832052 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:52.832845 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:52.837733 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:52.838160 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:53.838928 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:53.843842 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:53.844262 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:54.845163 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:54.850293 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:54.850719 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:55.851531 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:55.856470 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:55.856916 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:56.857774 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:56.862597 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:56.863023 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:57.863958 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:57.872071 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:57.872717 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:58.873570 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:38:58.878525 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:38:58.878912 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:38:59.902841 3527 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240502 21:38:59.903563 3527 tablet_server.cc:178] TabletServer@127.3.113.193:0 shutting down...
W20240502 21:38:59.922838 3702 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240502 21:38:59.927600 3702 consensus_peers.cc:485] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 -> Peer c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159): Couldn't send request to peer c5a1ad9a56f94dd3a5cb3e92c944ff1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:38:59.935189 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:38:59.935961 3527 tablet_replica.cc:330] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.936851 3527 raft_consensus.cc:2229] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:38:59.937971 3527 raft_consensus.cc:2258] T 925e68471ba24c42a502a8c6597f9e48 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.940796 3527 tablet_replica.cc:330] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.941380 3527 raft_consensus.cc:2229] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:38:59.942188 3527 raft_consensus.cc:2258] T 2cddf42b3217464e9ea06cf6bc13df79 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.944099 3527 tablet_replica.cc:330] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.944679 3527 raft_consensus.cc:2229] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.945142 3527 raft_consensus.cc:2258] T e05b127281f0418dbc30f30bdfed01f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.946910 3527 tablet_replica.cc:330] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.947376 3527 raft_consensus.cc:2229] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.947803 3527 raft_consensus.cc:2258] T 1a99e3a145c849a1926accca79b2dbc3 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.949483 3527 tablet_replica.cc:330] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.950024 3527 raft_consensus.cc:2229] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 LEADER]: Raft consensus shutting down.
I20240502 21:38:59.950822 3527 raft_consensus.cc:2258] T d61732cc5bb64686a0bad7d1c5a6d80d P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.952616 3527 tablet_replica.cc:330] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.953086 3527 raft_consensus.cc:2229] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.953508 3527 raft_consensus.cc:2258] T c10aff62cc254785a05e04ba677a4d34 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.955233 3527 tablet_replica.cc:330] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.955710 3527 raft_consensus.cc:2229] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.956135 3527 raft_consensus.cc:2258] T 6e8543cd0ff04c2598b1ea88638dd729 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.957762 3527 tablet_replica.cc:330] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.958184 3527 raft_consensus.cc:2229] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.958616 3527 raft_consensus.cc:2258] T 1e7792908e264a6bb90e85b9042524f2 P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:38:59.960073 3527 tablet_replica.cc:330] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d: stopping tablet replica
I20240502 21:38:59.960548 3527 raft_consensus.cc:2229] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:38:59.960945 3527 raft_consensus.cc:2258] T e113d6a5dcf34a4b9a3d585585215bda P c5a1ad9a56f94dd3a5cb3e92c944ff1d [term 3 FOLLOWER]: Raft consensus is shut down!
W20240502 21:38:59.972465 3776 consensus_peers.cc:485] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d -> Peer c5a1ad9a56f94dd3a5cb3e92c944ff1d (127.3.113.193:33159): Couldn't send request to peer c5a1ad9a56f94dd3a5cb3e92c944ff1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:00.050503 3527 tablet_server.cc:194] TabletServer@127.3.113.193:0 shutdown complete.
I20240502 21:39:00.067981 3527 tablet_server.cc:178] TabletServer@127.3.113.194:0 shutting down...
W20240502 21:39:00.076010 3776 consensus_peers.cc:485] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d -> Peer c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571): Couldn't send request to peer c0ab090735f1416987814c66f8cc9669. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:39:00.087412 3776 consensus_peers.cc:485] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d -> Peer c0ab090735f1416987814c66f8cc9669 (127.3.113.194:33571): Couldn't send request to peer c0ab090735f1416987814c66f8cc9669. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:00.096292 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:39:00.097131 3527 tablet_replica.cc:330] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.097810 3527 raft_consensus.cc:2229] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.098277 3527 raft_consensus.cc:2258] T 925e68471ba24c42a502a8c6597f9e48 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.100082 3527 tablet_replica.cc:330] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.100754 3527 raft_consensus.cc:2229] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.101258 3527 raft_consensus.cc:2258] T d61732cc5bb64686a0bad7d1c5a6d80d P c0ab090735f1416987814c66f8cc9669 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.103188 3527 tablet_replica.cc:330] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.103730 3527 raft_consensus.cc:2229] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.104193 3527 raft_consensus.cc:2258] T 2cddf42b3217464e9ea06cf6bc13df79 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.105899 3527 tablet_replica.cc:330] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.106355 3527 raft_consensus.cc:2229] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.106796 3527 raft_consensus.cc:2258] T e05b127281f0418dbc30f30bdfed01f2 P c0ab090735f1416987814c66f8cc9669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.108462 3527 tablet_replica.cc:330] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.108915 3527 raft_consensus.cc:2229] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.109719 3527 raft_consensus.cc:2258] T 1a99e3a145c849a1926accca79b2dbc3 P c0ab090735f1416987814c66f8cc9669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.111387 3527 tablet_replica.cc:330] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.111857 3527 raft_consensus.cc:2229] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.112815 3527 raft_consensus.cc:2258] T 6e8543cd0ff04c2598b1ea88638dd729 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.114416 3527 tablet_replica.cc:330] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.114876 3527 raft_consensus.cc:2229] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.115278 3527 raft_consensus.cc:2258] T 1e7792908e264a6bb90e85b9042524f2 P c0ab090735f1416987814c66f8cc9669 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.117213 3527 tablet_replica.cc:330] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.117692 3527 raft_consensus.cc:2229] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.118088 3527 raft_consensus.cc:2258] T c10aff62cc254785a05e04ba677a4d34 P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.119637 3527 tablet_replica.cc:330] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669: stopping tablet replica
I20240502 21:39:00.120059 3527 raft_consensus.cc:2229] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.120833 3527 raft_consensus.cc:2258] T e113d6a5dcf34a4b9a3d585585215bda P c0ab090735f1416987814c66f8cc9669 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.149216 3527 tablet_server.cc:194] TabletServer@127.3.113.194:0 shutdown complete.
I20240502 21:39:00.166858 3527 tablet_server.cc:178] TabletServer@127.3.113.195:0 shutting down...
I20240502 21:39:00.194509 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:39:00.195271 3527 tablet_replica.cc:330] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.195925 3527 raft_consensus.cc:2229] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.196498 3527 raft_consensus.cc:2258] T 925e68471ba24c42a502a8c6597f9e48 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.198230 3527 tablet_replica.cc:330] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.198771 3527 raft_consensus.cc:2229] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.199194 3527 raft_consensus.cc:2258] T d61732cc5bb64686a0bad7d1c5a6d80d P 2f60f6bcac494ebf8c632930c129d75d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.200834 3527 tablet_replica.cc:330] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.201313 3527 raft_consensus.cc:2229] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.201705 3527 raft_consensus.cc:2258] T 1a99e3a145c849a1926accca79b2dbc3 P 2f60f6bcac494ebf8c632930c129d75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.203495 3527 tablet_replica.cc:330] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.204012 3527 raft_consensus.cc:2229] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.204921 3527 raft_consensus.cc:2258] T c10aff62cc254785a05e04ba677a4d34 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.206662 3527 tablet_replica.cc:330] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.207137 3527 raft_consensus.cc:2229] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.207535 3527 raft_consensus.cc:2258] T 6e8543cd0ff04c2598b1ea88638dd729 P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.209215 3527 tablet_replica.cc:330] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.209676 3527 raft_consensus.cc:2229] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 5 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.210415 3527 raft_consensus.cc:2258] T 1e7792908e264a6bb90e85b9042524f2 P 2f60f6bcac494ebf8c632930c129d75d [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.211936 3527 tablet_replica.cc:330] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.212430 3527 raft_consensus.cc:2229] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.213200 3527 raft_consensus.cc:2258] T e05b127281f0418dbc30f30bdfed01f2 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.214782 3527 tablet_replica.cc:330] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.215277 3527 raft_consensus.cc:2229] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.215698 3527 raft_consensus.cc:2258] T 2cddf42b3217464e9ea06cf6bc13df79 P 2f60f6bcac494ebf8c632930c129d75d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.217223 3527 tablet_replica.cc:330] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d: stopping tablet replica
I20240502 21:39:00.217646 3527 raft_consensus.cc:2229] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:00.218034 3527 raft_consensus.cc:2258] T e113d6a5dcf34a4b9a3d585585215bda P 2f60f6bcac494ebf8c632930c129d75d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.246263 3527 tablet_server.cc:194] TabletServer@127.3.113.195:0 shutdown complete.
I20240502 21:39:00.263650 3527 master.cc:537] Master@127.3.113.254:34545 shutting down...
I20240502 21:39:00.466069 3527 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:39:00.466729 3527 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:00.467023 3527 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b983aa8a13146f89eeadcef2301ce75: stopping tablet replica
I20240502 21:39:00.485903 3527 master.cc:552] Master@127.3.113.254:34545 shutdown complete.
2f60f6bcac494ebf8c632930c129d75d 1
c0ab090735f1416987814c66f8cc9669 4
c5a1ad9a56f94dd3a5cb3e92c944ff1d 4
2f60f6bcac494ebf8c632930c129d75d 3
c0ab090735f1416987814c66f8cc9669 3
c5a1ad9a56f94dd3a5cb3e92c944ff1d 3
2f60f6bcac494ebf8c632930c129d75d 1
c0ab090735f1416987814c66f8cc9669 8
c5a1ad9a56f94dd3a5cb3e92c944ff1d 0
2f60f6bcac494ebf8c632930c129d75d 3
c0ab090735f1416987814c66f8cc9669 3
c5a1ad9a56f94dd3a5cb3e92c944ff1d 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31132 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240502 21:39:00.527086 3527 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.113.254:46397
I20240502 21:39:00.528229 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:00.536190 4056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.536568 4057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.537497 4059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.537935 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:00.538908 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:00.539095 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:00.539255 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685940539237 us; error 0 us; skew 500 ppm
I20240502 21:39:00.539779 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:00.548166 3527 webserver.cc:460] Webserver started at http://127.3.113.254:35407/ using document root <none> and password file <none>
I20240502 21:39:00.548759 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:00.548936 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:00.549252 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:00.550549 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/master-0-root/instance:
uuid: "aef88888bce9480fad615b0ac9103455"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.555245 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:39:00.558640 4064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.559371 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:39:00.559660 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/master-0-root
uuid: "aef88888bce9480fad615b0ac9103455"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.559954 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:00.580118 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:00.581537 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:00.620875 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.254:46397
I20240502 21:39:00.620957 4115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.254:46397 every 8 connection(s)
I20240502 21:39:00.624765 4116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:00.634902 4116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455: Bootstrap starting.
I20240502 21:39:00.639180 4116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:00.643332 4116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455: No bootstrap required, opened a new log
I20240502 21:39:00.645393 4116 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER }
I20240502 21:39:00.645866 4116 raft_consensus.cc:374] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:00.646044 4116 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef88888bce9480fad615b0ac9103455, State: Initialized, Role: FOLLOWER
I20240502 21:39:00.646492 4116 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER }
I20240502 21:39:00.646912 4116 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:39:00.647099 4116 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:39:00.647315 4116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:00.651188 4116 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER }
I20240502 21:39:00.651681 4116 leader_election.cc:302] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef88888bce9480fad615b0ac9103455; no voters:
I20240502 21:39:00.652761 4116 leader_election.cc:288] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:39:00.653196 4119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:00.654556 4119 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 1 LEADER]: Becoming Leader. State: Replica: aef88888bce9480fad615b0ac9103455, State: Running, Role: LEADER
I20240502 21:39:00.655282 4119 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER }
I20240502 21:39:00.655719 4116 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:39:00.658612 4121 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef88888bce9480fad615b0ac9103455. Latest consensus state: current_term: 1 leader_uuid: "aef88888bce9480fad615b0ac9103455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER } }
I20240502 21:39:00.658543 4120 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aef88888bce9480fad615b0ac9103455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef88888bce9480fad615b0ac9103455" member_type: VOTER } }
I20240502 21:39:00.659339 4121 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [sys.catalog]: This master's current role is: LEADER
I20240502 21:39:00.659438 4120 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [sys.catalog]: This master's current role is: LEADER
I20240502 21:39:00.662292 4124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:39:00.670608 4124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:39:00.671751 3527 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:39:00.679387 4124 catalog_manager.cc:1347] Generated new cluster ID: dc26167a85694f4f875a89c405ad07f1
I20240502 21:39:00.679737 4124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:39:00.706537 4124 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:39:00.707816 4124 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:39:00.723445 4124 catalog_manager.cc:5857] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455: Generated new TSK 0
I20240502 21:39:00.724114 4124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:39:00.738724 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:00.744894 4137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.746431 4138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.748510 4140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.748768 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:00.749682 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:00.749889 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:00.750041 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685940750023 us; error 0 us; skew 500 ppm
I20240502 21:39:00.750598 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:00.752969 3527 webserver.cc:460] Webserver started at http://127.3.113.193:44519/ using document root <none> and password file <none>
I20240502 21:39:00.753451 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:00.753633 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:00.753892 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:00.754984 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-0-root/instance:
uuid: "9b5a176db1134fc9a21391052977e0e9"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.759832 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:39:00.763450 4145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.764343 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:39:00.764724 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
uuid: "9b5a176db1134fc9a21391052977e0e9"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.765178 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:00.777770 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:00.779052 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:00.780671 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:00.783079 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:00.783282 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.783515 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:00.783716 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.824050 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.193:35383
I20240502 21:39:00.824160 4208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.193:35383 every 8 connection(s)
I20240502 21:39:00.828765 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:00.836979 4213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.839017 4214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.841987 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:00.842214 4209 heartbeater.cc:347] Connected to a master server at 127.3.113.254:46397
I20240502 21:39:00.842669 4209 heartbeater.cc:464] Registering TS with master...
W20240502 21:39:00.842855 4216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.843502 4209 heartbeater.cc:511] Master 127.3.113.254:46397 requested a full tablet report, sending...
I20240502 21:39:00.843737 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:00.843974 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:00.844128 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685940844112 us; error 0 us; skew 500 ppm
I20240502 21:39:00.844853 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:00.845544 4081 ts_manager.cc:194] Registered new tserver with Master: 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383)
I20240502 21:39:00.847229 4081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60950
I20240502 21:39:00.849892 3527 webserver.cc:460] Webserver started at http://127.3.113.194:46719/ using document root <none> and password file <none>
I20240502 21:39:00.850410 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:00.850600 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:00.850853 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:00.851969 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-1-root/instance:
uuid: "3c00fe81035148709139976e14482b50"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.856420 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 21:39:00.859793 4221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.860595 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:39:00.860877 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
uuid: "3c00fe81035148709139976e14482b50"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.861218 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:00.877313 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:00.878587 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:00.880024 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:00.882422 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:00.882627 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.882869 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:00.883028 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.922341 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.194:39657
I20240502 21:39:00.922439 4285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.194:39657 every 8 connection(s)
I20240502 21:39:00.926997 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:00.933871 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:00.935658 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.939181 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:00.939934 4286 heartbeater.cc:347] Connected to a master server at 127.3.113.254:46397
W20240502 21:39:00.940152 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:00.940415 4286 heartbeater.cc:464] Registering TS with master...
I20240502 21:39:00.941125 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240502 21:39:00.941254 4286 heartbeater.cc:511] Master 127.3.113.254:46397 requested a full tablet report, sending...
W20240502 21:39:00.941366 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:00.941650 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685940941629 us; error 0 us; skew 500 ppm
I20240502 21:39:00.942354 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:00.943521 4081 ts_manager.cc:194] Registered new tserver with Master: 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:00.945240 3527 webserver.cc:460] Webserver started at http://127.3.113.195:46559/ using document root <none> and password file <none>
I20240502 21:39:00.945468 4081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60962
I20240502 21:39:00.945927 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:00.946199 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:00.946527 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:00.947764 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-2-root/instance:
uuid: "bebef7198ff84fc191d1f0870cbc2237"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.952179 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20240502 21:39:00.955312 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.956137 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:39:00.956496 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
uuid: "bebef7198ff84fc191d1f0870cbc2237"
format_stamp: "Formatted at 2024-05-02 21:39:00 on dist-test-slave-pddb"
I20240502 21:39:00.956830 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714685909228215-3527-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:00.972121 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:00.973393 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:00.974870 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:00.977079 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:00.977283 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:00.977525 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:00.977687 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:01.017463 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.195:37083
I20240502 21:39:01.017571 4359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.195:37083 every 8 connection(s)
I20240502 21:39:01.030499 4360 heartbeater.cc:347] Connected to a master server at 127.3.113.254:46397
I20240502 21:39:01.030853 4360 heartbeater.cc:464] Registering TS with master...
I20240502 21:39:01.031567 4360 heartbeater.cc:511] Master 127.3.113.254:46397 requested a full tablet report, sending...
I20240502 21:39:01.033411 4081 ts_manager.cc:194] Registered new tserver with Master: bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.033982 3527 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013462762s
I20240502 21:39:01.035108 4081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60976
I20240502 21:39:01.037027 3527 test_util.cc:268] Using random seed: 1852487628
I20240502 21:39:01.059877 4081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60986:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:39:01.063194 4081 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:39:01.129402 4248 tablet_service.cc:1467] Processing CreateTablet for tablet 94f6ea4fdc1345fd908811a8ca892e32 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 21:39:01.130980 4248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f6ea4fdc1345fd908811a8ca892e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.130849 4249 tablet_service.cc:1467] Processing CreateTablet for tablet 5b75edea9b274b8a82e195f0a5dde101 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 21:39:01.132266 4249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b75edea9b274b8a82e195f0a5dde101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.133291 4247 tablet_service.cc:1467] Processing CreateTablet for tablet c1466665aa664ba7acca06db61caf818 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 21:39:01.134593 4247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1466665aa664ba7acca06db61caf818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.137568 4246 tablet_service.cc:1467] Processing CreateTablet for tablet d0fd256f23a745608bc8a9cb1130675f (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 21:39:01.138919 4246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fd256f23a745608bc8a9cb1130675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.141543 4245 tablet_service.cc:1467] Processing CreateTablet for tablet 361ddfb10f164992a4837b435bd0d69d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 21:39:01.142880 4245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ddfb10f164992a4837b435bd0d69d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.144052 4171 tablet_service.cc:1467] Processing CreateTablet for tablet d0fd256f23a745608bc8a9cb1130675f (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 21:39:01.144146 4170 tablet_service.cc:1467] Processing CreateTablet for tablet 361ddfb10f164992a4837b435bd0d69d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 21:39:01.145519 4171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fd256f23a745608bc8a9cb1130675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.146181 4169 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60a625cb6242bf98c9f02d2ed4569d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 21:39:01.145524 4244 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60a625cb6242bf98c9f02d2ed4569d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 21:39:01.146983 4324 tablet_service.cc:1467] Processing CreateTablet for tablet 94f6ea4fdc1345fd908811a8ca892e32 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 21:39:01.148067 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60a625cb6242bf98c9f02d2ed4569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.148154 4244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60a625cb6242bf98c9f02d2ed4569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.149178 4170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ddfb10f164992a4837b435bd0d69d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.149571 4323 tablet_service.cc:1467] Processing CreateTablet for tablet c1466665aa664ba7acca06db61caf818 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 21:39:01.150787 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 94f6ea4fdc1345fd908811a8ca892e32 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240502 21:39:01.152112 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f6ea4fdc1345fd908811a8ca892e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.151911 4322 tablet_service.cc:1467] Processing CreateTablet for tablet d0fd256f23a745608bc8a9cb1130675f (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240502 21:39:01.154240 4321 tablet_service.cc:1467] Processing CreateTablet for tablet 361ddfb10f164992a4837b435bd0d69d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240502 21:39:01.154464 4174 tablet_service.cc:1467] Processing CreateTablet for tablet 5b75edea9b274b8a82e195f0a5dde101 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 21:39:01.155583 4174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b75edea9b274b8a82e195f0a5dde101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.145557 4325 tablet_service.cc:1467] Processing CreateTablet for tablet 5b75edea9b274b8a82e195f0a5dde101 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240502 21:39:01.149571 4172 tablet_service.cc:1467] Processing CreateTablet for tablet c1466665aa664ba7acca06db61caf818 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240502 21:39:01.157903 4172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1466665aa664ba7acca06db61caf818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.161231 4324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f6ea4fdc1345fd908811a8ca892e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.163268 4321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ddfb10f164992a4837b435bd0d69d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.164651 4322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fd256f23a745608bc8a9cb1130675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.165761 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1466665aa664ba7acca06db61caf818. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.166992 4325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b75edea9b274b8a82e195f0a5dde101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.196219 4320 tablet_service.cc:1467] Processing CreateTablet for tablet 5a60a625cb6242bf98c9f02d2ed4569d (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240502 21:39:01.197670 4320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a60a625cb6242bf98c9f02d2ed4569d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.218919 4168 tablet_service.cc:1467] Processing CreateTablet for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 21:39:01.219627 4243 tablet_service.cc:1467] Processing CreateTablet for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 21:39:01.220449 4168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.221037 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.230993 4319 tablet_service.cc:1467] Processing CreateTablet for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240502 21:39:01.232409 4319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0019d4ae8ffb4dc68e1a180f9c7e8d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.236265 4167 tablet_service.cc:1467] Processing CreateTablet for tablet 57eeae8767c747bfa7a79c8185bc25f0 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 21:39:01.237715 4167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57eeae8767c747bfa7a79c8185bc25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.241591 4242 tablet_service.cc:1467] Processing CreateTablet for tablet 57eeae8767c747bfa7a79c8185bc25f0 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 21:39:01.242977 4242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57eeae8767c747bfa7a79c8185bc25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.251556 4318 tablet_service.cc:1467] Processing CreateTablet for tablet 57eeae8767c747bfa7a79c8185bc25f0 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240502 21:39:01.252990 4318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57eeae8767c747bfa7a79c8185bc25f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.257560 4166 tablet_service.cc:1467] Processing CreateTablet for tablet e58c31ea9ec54aad8d051faca95b94e9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 21:39:01.258947 4166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58c31ea9ec54aad8d051faca95b94e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.261651 4317 tablet_service.cc:1467] Processing CreateTablet for tablet e58c31ea9ec54aad8d051faca95b94e9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 21:39:01.265398 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8e35b25514dfd91cc7c85c67147a9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 21:39:01.277469 4243 tablet_service.cc:1467] Processing CreateTablet for tablet e58c31ea9ec54aad8d051faca95b94e9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240502 21:39:01.279289 4243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58c31ea9ec54aad8d051faca95b94e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.286372 4375 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.301165 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8e35b25514dfd91cc7c85c67147a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.304991 4317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58c31ea9ec54aad8d051faca95b94e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.307546 4376 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.312101 4376 tablet_bootstrap.cc:654] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.314087 4246 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8e35b25514dfd91cc7c85c67147a9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 21:39:01.315425 4246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8e35b25514dfd91cc7c85c67147a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.317370 4375 tablet_bootstrap.cc:654] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.317770 4377 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.320247 4320 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8e35b25514dfd91cc7c85c67147a9 (DEFAULT_TABLE table=test-workload [id=c809729b9d8a435aa025b2c453f64bdb]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240502 21:39:01.323372 4320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8e35b25514dfd91cc7c85c67147a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:01.333978 4377 tablet_bootstrap.cc:654] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.353166 4375 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.353963 4377 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.351451 4376 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.354445 4377 ts_tablet_manager.cc:1367] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.006s
I20240502 21:39:01.354610 4376 ts_tablet_manager.cc:1367] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.015s
I20240502 21:39:01.355919 4375 ts_tablet_manager.cc:1367] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.070s user 0.021s sys 0.008s
I20240502 21:39:01.357255 4377 raft_consensus.cc:348] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.357467 4376 raft_consensus.cc:348] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.358192 4377 raft_consensus.cc:374] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.358273 4376 raft_consensus.cc:374] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.359218 4376 raft_consensus.cc:729] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.359992 4376 consensus_queue.cc:260] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.358646 4377 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.362126 4377 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.362823 4375 raft_consensus.cc:348] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.363694 4375 raft_consensus.cc:374] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.364048 4375 raft_consensus.cc:729] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.365015 4360 heartbeater.cc:503] Master 127.3.113.254:46397 was elected leader, sending a full tablet report...
I20240502 21:39:01.365298 4377 ts_tablet_manager.cc:1398] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20240502 21:39:01.366339 4377 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.370668 4375 consensus_queue.cc:260] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.372350 4377 tablet_bootstrap.cc:654] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.376410 4209 heartbeater.cc:503] Master 127.3.113.254:46397 was elected leader, sending a full tablet report...
I20240502 21:39:01.376964 4376 ts_tablet_manager.cc:1398] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.022s user 0.012s sys 0.010s
I20240502 21:39:01.378242 4376 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.386577 4376 tablet_bootstrap.cc:654] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.388836 4377 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.389400 4377 ts_tablet_manager.cc:1367] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.008s
I20240502 21:39:01.392217 4377 raft_consensus.cc:348] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.393189 4377 raft_consensus.cc:374] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.393493 4377 raft_consensus.cc:729] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.394383 4377 consensus_queue.cc:260] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.396914 4377 ts_tablet_manager.cc:1398] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 21:39:01.397940 4377 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.405143 4377 tablet_bootstrap.cc:654] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.415390 4375 ts_tablet_manager.cc:1398] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.059s user 0.018s sys 0.029s
I20240502 21:39:01.418215 4286 heartbeater.cc:503] Master 127.3.113.254:46397 was elected leader, sending a full tablet report...
I20240502 21:39:01.419414 4376 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.420073 4375 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.418205 4377 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.419973 4376 ts_tablet_manager.cc:1367] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.028s
I20240502 21:39:01.420862 4377 ts_tablet_manager.cc:1367] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20240502 21:39:01.423538 4376 raft_consensus.cc:348] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.423831 4377 raft_consensus.cc:348] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.424523 4376 raft_consensus.cc:374] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.424875 4377 raft_consensus.cc:374] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.424973 4376 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.419983 4383 raft_consensus.cc:482] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.425338 4377 raft_consensus.cc:729] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.425761 4383 raft_consensus.cc:504] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.427326 4375 tablet_bootstrap.cc:654] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.427985 4383 leader_election.cc:288] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.428857 4376 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.431304 4376 ts_tablet_manager.cc:1398] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:39:01.432252 4376 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.426177 4377 consensus_queue.cc:260] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.437299 4377 ts_tablet_manager.cc:1398] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 21:39:01.438405 4377 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.439353 4376 tablet_bootstrap.cc:654] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.445540 4377 tablet_bootstrap.cc:654] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.465628 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:01.471247 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:01.472419 4335 raft_consensus.cc:2454] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 0.
I20240502 21:39:01.475921 4224 leader_election.cc:302] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.476886 4383 raft_consensus.cc:2781] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 21:39:01.476996 4224 leader_election.cc:341] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.477356 4383 raft_consensus.cc:482] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.477754 4383 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.478648 4375 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.479147 4375 ts_tablet_manager.cc:1367] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.007s
I20240502 21:39:01.482625 4376 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.483211 4376 ts_tablet_manager.cc:1367] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.039s
I20240502 21:39:01.485122 4383 raft_consensus.cc:504] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.486150 4376 raft_consensus.cc:348] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.487001 4376 raft_consensus.cc:374] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.487362 4376 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.487975 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:01.487713 4375 raft_consensus.cc:348] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.488226 4376 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.488793 4375 raft_consensus.cc:374] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.489353 4375 raft_consensus.cc:729] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
W20240502 21:39:01.489444 4224 leader_election.cc:341] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.490960 4376 ts_tablet_manager.cc:1398] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:39:01.490913 4375 consensus_queue.cc:260] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.491402 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:01.491972 4335 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.492089 4376 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.493939 4375 ts_tablet_manager.cc:1398] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20240502 21:39:01.494112 4377 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.494661 4377 ts_tablet_manager.cc:1367] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.056s user 0.025s sys 0.020s
I20240502 21:39:01.497397 4377 raft_consensus.cc:348] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.498191 4377 raft_consensus.cc:374] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.498281 4376 tablet_bootstrap.cc:654] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.498664 4377 raft_consensus.cc:729] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.499595 4377 consensus_queue.cc:260] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.502314 4375 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.507119 4377 ts_tablet_manager.cc:1398] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240502 21:39:01.508211 4377 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.512465 4375 tablet_bootstrap.cc:654] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.515990 4381 raft_consensus.cc:482] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.516841 4335 raft_consensus.cc:2454] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 1.
I20240502 21:39:01.516554 4381 raft_consensus.cc:504] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.519508 4224 leader_election.cc:302] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters: 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:01.520658 4391 raft_consensus.cc:2781] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.521674 4381 leader_election.cc:288] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.522929 4381 raft_consensus.cc:482] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.523496 4381 raft_consensus.cc:504] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.523981 4383 leader_election.cc:288] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.525797 4381 leader_election.cc:288] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.526252 4391 raft_consensus.cc:686] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:01.527563 4391 consensus_queue.cc:237] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.532792 4377 tablet_bootstrap.cc:654] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.540977 4375 tablet_bootstrap.cc:492] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.541512 4375 ts_tablet_manager.cc:1367] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.012s
I20240502 21:39:01.545024 4377 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.545632 4377 ts_tablet_manager.cc:1367] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20240502 21:39:01.546408 4376 tablet_bootstrap.cc:492] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.546907 4376 ts_tablet_manager.cc:1367] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.055s user 0.020s sys 0.007s
I20240502 21:39:01.548734 4377 raft_consensus.cc:348] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.549656 4377 raft_consensus.cc:374] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.549526 4375 raft_consensus.cc:348] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.550048 4377 raft_consensus.cc:729] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.550357 4375 raft_consensus.cc:374] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.551816 4375 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.550029 4078 catalog_manager.cc:5484] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:01.552956 4375 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.553665 4376 raft_consensus.cc:348] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.555264 4375 ts_tablet_manager.cc:1398] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:39:01.555783 4376 raft_consensus.cc:374] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.556104 4376 raft_consensus.cc:729] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.556537 4375 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.556988 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:01.551292 4377 consensus_queue.cc:260] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.557708 4335 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 0.
I20240502 21:39:01.557547 4376 consensus_queue.cc:260] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.558851 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:01.559191 4377 ts_tablet_manager.cc:1398] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:39:01.559521 4334 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 0.
I20240502 21:39:01.562080 4377 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.562918 4375 tablet_bootstrap.cc:654] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.566273 4149 leader_election.cc:302] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.567979 4377 tablet_bootstrap.cc:654] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.570053 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.571856 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.572821 4394 raft_consensus.cc:2781] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.573372 4394 raft_consensus.cc:482] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.573784 4394 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.574560 4376 ts_tablet_manager.cc:1398] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20240502 21:39:01.575922 4376 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.572857 4259 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 0.
I20240502 21:39:01.567647 4149 leader_election.cc:302] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.579946 4381 raft_consensus.cc:2781] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 21:39:01.580071 4150 leader_election.cc:341] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c00fe81035148709139976e14482b50 (127.3.113.194:39657): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.580526 4381 raft_consensus.cc:482] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.581030 4381 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.596930 4394 raft_consensus.cc:504] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.599718 4394 leader_election.cc:288] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.600570 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.600792 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:01.597808 4381 raft_consensus.cc:504] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.601642 4334 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.601174 4259 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.603642 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.604246 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:01.604874 4335 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.605192 4375 tablet_bootstrap.cc:492] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.605669 4375 ts_tablet_manager.cc:1367] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.027s
I20240502 21:39:01.607753 4334 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 1.
I20240502 21:39:01.608445 4381 leader_election.cc:288] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.596192 4376 tablet_bootstrap.cc:654] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.608373 4375 raft_consensus.cc:348] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.611887 4375 raft_consensus.cc:374] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:39:01.612541 4150 leader_election.cc:341] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c00fe81035148709139976e14482b50 (127.3.113.194:39657): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.612788 4375 raft_consensus.cc:729] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.612727 4149 leader_election.cc:302] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.611416 4335 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 1.
I20240502 21:39:01.613714 4375 consensus_queue.cc:260] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.614166 4259 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 1.
I20240502 21:39:01.614488 4149 leader_election.cc:302] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters: 3c00fe81035148709139976e14482b50
I20240502 21:39:01.615621 4377 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.616206 4375 ts_tablet_manager.cc:1398] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20240502 21:39:01.616807 4377 ts_tablet_manager.cc:1367] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.023s
I20240502 21:39:01.617302 4375 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.620088 4377 raft_consensus.cc:348] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.620867 4377 raft_consensus.cc:374] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.621167 4377 raft_consensus.cc:729] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.621747 4376 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.622241 4376 ts_tablet_manager.cc:1367] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.001s
I20240502 21:39:01.621943 4377 consensus_queue.cc:260] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.623854 4375 tablet_bootstrap.cc:654] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.625042 4376 raft_consensus.cc:348] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.625777 4376 raft_consensus.cc:374] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.626060 4376 raft_consensus.cc:729] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.626983 4377 ts_tablet_manager.cc:1398] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 21:39:01.627346 4394 raft_consensus.cc:2781] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.626906 4376 consensus_queue.cc:260] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.627943 4377 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.628504 4381 raft_consensus.cc:2781] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.644877 4377 tablet_bootstrap.cc:654] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.648286 4382 raft_consensus.cc:482] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.649021 4382 raft_consensus.cc:504] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.650434 4405 raft_consensus.cc:482] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.651037 4405 raft_consensus.cc:504] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.656145 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.656896 4405 leader_election.cc:288] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.658784 4375 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.658836 4376 ts_tablet_manager.cc:1398] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.036s user 0.012s sys 0.006s
I20240502 21:39:01.659708 4375 ts_tablet_manager.cc:1367] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.009s
I20240502 21:39:01.660480 4376 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.661134 4382 leader_election.cc:288] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.661394 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:01.663246 4335 raft_consensus.cc:2379] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b5a176db1134fc9a21391052977e0e9 in current term 1: Already voted for candidate 3c00fe81035148709139976e14482b50 in this term.
I20240502 21:39:01.663380 4377 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.663933 4377 ts_tablet_manager.cc:1367] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.005s
I20240502 21:39:01.664546 4149 leader_election.cc:302] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9; no voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:01.665241 4375 raft_consensus.cc:348] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.666172 4375 raft_consensus.cc:374] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.666659 4375 raft_consensus.cc:729] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
W20240502 21:39:01.667753 4129 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:01.668344 4405 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.668788 4376 tablet_bootstrap.cc:654] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.667753 4375 consensus_queue.cc:260] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.669538 4394 raft_consensus.cc:686] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:01.670414 4394 consensus_queue.cc:237] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.672152 4381 raft_consensus.cc:686] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:01.673048 4381 consensus_queue.cc:237] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.678402 4375 ts_tablet_manager.cc:1398] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:39:01.679303 4377 raft_consensus.cc:348] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.680390 4377 raft_consensus.cc:374] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.680779 4377 raft_consensus.cc:729] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.682229 4405 raft_consensus.cc:2726] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:01.687165 4375 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.686627 4377 consensus_queue.cc:260] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.693243 4377 ts_tablet_manager.cc:1398] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.029s user 0.005s sys 0.005s
I20240502 21:39:01.693900 4375 tablet_bootstrap.cc:654] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.694308 4377 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.695605 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
W20240502 21:39:01.699792 4300 leader_election.cc:341] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.700057 4382 raft_consensus.cc:482] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.700785 4382 raft_consensus.cc:504] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.704936 4382 leader_election.cc:288] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.705297 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
W20240502 21:39:01.707181 4300 leader_election.cc:341] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.708629 4376 tablet_bootstrap.cc:492] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.709152 4376 ts_tablet_manager.cc:1367] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.049s user 0.030s sys 0.008s
I20240502 21:39:01.710289 4377 tablet_bootstrap.cc:654] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.710702 4375 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.712024 4376 raft_consensus.cc:348] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.712853 4376 raft_consensus.cc:374] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.713198 4376 raft_consensus.cc:729] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.713399 4078 catalog_manager.cc:5484] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:01.714246 4376 consensus_queue.cc:260] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.716486 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.716704 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.717149 4259 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 0.
I20240502 21:39:01.717244 4258 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 0.
I20240502 21:39:01.717340 4376 ts_tablet_manager.cc:1398] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:39:01.718611 4375 ts_tablet_manager.cc:1367] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.005s
I20240502 21:39:01.718499 4301 leader_election.cc:302] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters: 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:01.718777 4376 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.719805 4301 leader_election.cc:302] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters: 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:01.720851 4375 raft_consensus.cc:348] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.721757 4375 raft_consensus.cc:374] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.722077 4375 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.722616 4417 raft_consensus.cc:2781] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.723083 4417 raft_consensus.cc:482] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.723573 4417 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.723013 4375 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.724237 4382 raft_consensus.cc:2781] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.724651 4382 raft_consensus.cc:482] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.724983 4382 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.725483 4376 tablet_bootstrap.cc:654] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.731042 4417 raft_consensus.cc:504] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.732939 4375 ts_tablet_manager.cc:1398] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20240502 21:39:01.734146 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
W20240502 21:39:01.739625 4300 leader_election.cc:341] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.740590 4375 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.741700 4382 raft_consensus.cc:504] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.743804 4080 catalog_manager.cc:5484] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:01.745528 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:01.746490 4184 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.747960 4417 leader_election.cc:288] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.747166 4375 tablet_bootstrap.cc:654] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.752807 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.753432 4258 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.758163 4184 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.759776 4300 leader_election.cc:302] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.760584 4382 leader_election.cc:288] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.760886 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.761898 4259 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.761065 4417 raft_consensus.cc:2781] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.764137 4417 raft_consensus.cc:686] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:01.766856 4377 tablet_bootstrap.cc:492] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.765349 4417 consensus_queue.cc:237] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.767589 4377 ts_tablet_manager.cc:1367] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.073s user 0.016s sys 0.003s
I20240502 21:39:01.769924 4259 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.770571 4377 raft_consensus.cc:348] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.771354 4377 raft_consensus.cc:374] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.771689 4377 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.771948 4258 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.773180 4301 leader_election.cc:302] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters: 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:01.774032 4417 raft_consensus.cc:2781] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.773792 4377 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.774739 4417 raft_consensus.cc:686] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:01.775241 4376 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.775743 4376 ts_tablet_manager.cc:1367] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.057s user 0.031s sys 0.023s
I20240502 21:39:01.776271 4377 ts_tablet_manager.cc:1398] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:39:01.775882 4417 consensus_queue.cc:237] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.777315 4377 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.778470 4376 raft_consensus.cc:348] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.779211 4376 raft_consensus.cc:374] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.779508 4376 raft_consensus.cc:729] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.779533 4080 catalog_manager.cc:5484] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 0 to 1, leader changed from <none> to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:01.780464 4376 consensus_queue.cc:260] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.783344 4377 tablet_bootstrap.cc:654] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.785151 4376 ts_tablet_manager.cc:1398] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:39:01.786144 4376 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.794415 4391 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.795140 4376 tablet_bootstrap.cc:654] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.795061 4391 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.797755 4391 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.799103 4382 raft_consensus.cc:482] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.799782 4382 raft_consensus.cc:504] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.800742 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:01.801447 4335 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 0.
I20240502 21:39:01.802875 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:01.803485 4184 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 0.
I20240502 21:39:01.804603 4224 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.805047 4375 tablet_bootstrap.cc:492] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.805670 4375 ts_tablet_manager.cc:1367] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.008s
I20240502 21:39:01.806416 4391 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.807497 4391 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.807950 4391 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.808038 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:01.808424 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.802255 4382 leader_election.cc:288] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.809979 4184 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 0.
I20240502 21:39:01.817723 4300 leader_election.cc:302] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.818881 4382 raft_consensus.cc:2781] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.818903 4391 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.819561 4382 raft_consensus.cc:482] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.820050 4382 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.819970 4080 catalog_manager.cc:5484] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 0 to 1, leader changed from <none> to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:01.821929 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:01.822527 4184 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:01.826050 4301 leader_election.cc:341] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c00fe81035148709139976e14482b50 (127.3.113.194:39657): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.827404 4375 raft_consensus.cc:348] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.828441 4375 raft_consensus.cc:374] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.828809 4375 raft_consensus.cc:729] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.835963 4184 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 1.
I20240502 21:39:01.836930 4375 consensus_queue.cc:260] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.837678 4224 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:01.830456 4417 raft_consensus.cc:482] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:01.839015 4391 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:01.841564 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:01.841822 4375 ts_tablet_manager.cc:1398] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.036s user 0.008s sys 0.001s
I20240502 21:39:01.842182 4335 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.842877 4375 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.839717 4417 raft_consensus.cc:504] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.844465 4391 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.843957 4376 tablet_bootstrap.cc:492] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.844838 4382 raft_consensus.cc:504] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.845839 4417 leader_election.cc:288] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.847576 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:01.847587 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:01.845995 4391 raft_consensus.cc:686] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:01.848755 4376 ts_tablet_manager.cc:1367] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.063s user 0.024s sys 0.018s
W20240502 21:39:01.848912 4301 leader_election.cc:341] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c00fe81035148709139976e14482b50 (127.3.113.194:39657): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:01.849366 4391 consensus_queue.cc:237] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.850828 4377 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.851274 4377 ts_tablet_manager.cc:1367] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.074s user 0.015s sys 0.016s
I20240502 21:39:01.851961 4376 raft_consensus.cc:348] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.852797 4376 raft_consensus.cc:374] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.853318 4376 raft_consensus.cc:729] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.853941 4335 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 1.
I20240502 21:39:01.854187 4376 consensus_queue.cc:260] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.854238 4377 raft_consensus.cc:348] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.855361 4377 raft_consensus.cc:374] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.848340 4184 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 0.
I20240502 21:39:01.856530 4377 raft_consensus.cc:729] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.857322 4376 ts_tablet_manager.cc:1398] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 21:39:01.857323 4377 consensus_queue.cc:260] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.858472 4376 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.858608 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:01.859242 4183 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.859757 4377 ts_tablet_manager.cc:1398] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 21:39:01.860622 4377 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: Bootstrap starting.
I20240502 21:39:01.857249 4382 leader_election.cc:288] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.862699 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.863408 4259 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.868584 4300 leader_election.cc:302] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters: 3c00fe81035148709139976e14482b50
I20240502 21:39:01.870150 4417 raft_consensus.cc:2781] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:01.870653 4417 raft_consensus.cc:482] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:01.871070 4417 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.874092 4376 tablet_bootstrap.cc:654] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.875694 4377 tablet_bootstrap.cc:654] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.876365 4375 tablet_bootstrap.cc:654] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.888806 4375 tablet_bootstrap.cc:492] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.889470 4375 ts_tablet_manager.cc:1367] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.010s
I20240502 21:39:01.890245 4376 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.890813 4376 ts_tablet_manager.cc:1367] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.012s
I20240502 21:39:01.893299 4375 raft_consensus.cc:348] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.894253 4375 raft_consensus.cc:374] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.894470 4375 raft_consensus.cc:729] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.893522 4376 raft_consensus.cc:348] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.895447 4376 raft_consensus.cc:374] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.895152 4375 consensus_queue.cc:260] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.895756 4376 raft_consensus.cc:729] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.896466 4377 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: No bootstrap required, opened a new log
I20240502 21:39:01.896966 4377 ts_tablet_manager.cc:1367] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.025s
I20240502 21:39:01.896628 4376 consensus_queue.cc:260] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.897569 4375 ts_tablet_manager.cc:1398] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:39:01.898466 4375 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.898955 4376 ts_tablet_manager.cc:1398] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20240502 21:39:01.899837 4376 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: Bootstrap starting.
I20240502 21:39:01.899747 4377 raft_consensus.cc:348] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.900653 4377 raft_consensus.cc:374] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.900997 4377 raft_consensus.cc:729] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.901855 4377 consensus_queue.cc:260] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.903828 4183 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.906409 4375 tablet_bootstrap.cc:654] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.907059 4300 leader_election.cc:302] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.908031 4382 raft_consensus.cc:2781] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.904944 4417 raft_consensus.cc:504] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.909919 4417 leader_election.cc:288] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:01.911437 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:01.912248 4183 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.906402 4376 tablet_bootstrap.cc:654] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.915377 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:01.916039 4258 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:01.904028 4377 ts_tablet_manager.cc:1398] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:39:01.900244 4259 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.926450 4391 consensus_queue.cc:1035] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:01.927755 4080 catalog_manager.cc:5484] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:01.939783 4183 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.942322 4382 raft_consensus.cc:686] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:01.949864 4375 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:01.949760 4382 consensus_queue.cc:237] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.950420 4375 ts_tablet_manager.cc:1367] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.005s
I20240502 21:39:01.944486 4258 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 1.
I20240502 21:39:01.954191 4375 raft_consensus.cc:348] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.955076 4375 raft_consensus.cc:374] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.953758 4301 leader_election.cc:302] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:01.955503 4375 raft_consensus.cc:729] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.956449 4375 consensus_queue.cc:260] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.959442 4428 raft_consensus.cc:2781] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:01.960235 4428 raft_consensus.cc:686] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:01.966562 4375 ts_tablet_manager.cc:1398] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:39:01.969049 4375 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: Bootstrap starting.
I20240502 21:39:01.969626 4376 tablet_bootstrap.cc:492] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: No bootstrap required, opened a new log
I20240502 21:39:01.961558 4428 consensus_queue.cc:237] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.970849 4376 ts_tablet_manager.cc:1367] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.001s
I20240502 21:39:01.973608 4396 consensus_queue.cc:1035] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:01.974249 4376 raft_consensus.cc:348] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.975126 4376 raft_consensus.cc:374] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:01.975477 4376 raft_consensus.cc:729] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Initialized, Role: FOLLOWER
I20240502 21:39:01.979166 4375 tablet_bootstrap.cc:654] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:01.978873 4376 consensus_queue.cc:260] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:01.991559 4376 ts_tablet_manager.cc:1398] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: Time spent starting tablet: real 0.020s user 0.001s sys 0.005s
I20240502 21:39:01.999723 4080 catalog_manager.cc:5484] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 0 to 1, leader changed from <none> to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:02.001138 4375 tablet_bootstrap.cc:492] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: No bootstrap required, opened a new log
I20240502 21:39:02.001660 4375 ts_tablet_manager.cc:1367] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.001s
I20240502 21:39:02.005474 4375 raft_consensus.cc:348] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.006206 4375 raft_consensus.cc:374] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:02.006564 4375 raft_consensus.cc:729] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Initialized, Role: FOLLOWER
I20240502 21:39:02.007309 4375 consensus_queue.cc:260] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.009436 4375 ts_tablet_manager.cc:1398] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 21:39:02.018538 4080 catalog_manager.cc:5484] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 0 to 1, leader changed from <none> to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:02.043437 4394 raft_consensus.cc:482] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:02.044117 4394 raft_consensus.cc:504] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.045734 4394 leader_election.cc:288] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.046600 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:02.046741 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:02.047150 4258 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 0.
I20240502 21:39:02.047326 4335 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 0.
I20240502 21:39:02.048036 4150 leader_election.cc:302] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.048820 4394 raft_consensus.cc:2781] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:02.049160 4394 raft_consensus.cc:482] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:02.049415 4394 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.054430 4394 raft_consensus.cc:504] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.054975 4396 raft_consensus.cc:482] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:02.055603 4381 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:02.055548 4396 raft_consensus.cc:504] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.056510 4394 leader_election.cc:288] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.058681 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.058992 4334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:02.057874 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.059273 4258 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.059762 4335 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.059744 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:02.057897 4396 leader_election.cc:288] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.059780 4334 raft_consensus.cc:2379] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c00fe81035148709139976e14482b50 in current term 1: Already voted for candidate 9b5a176db1134fc9a21391052977e0e9 in this term.
I20240502 21:39:02.061806 4224 leader_election.cc:302] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c00fe81035148709139976e14482b50; no voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:02.063071 4396 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.067579 4335 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 1.
I20240502 21:39:02.068691 4149 leader_election.cc:302] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:02.069629 4394 raft_consensus.cc:2781] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:02.070338 4394 raft_consensus.cc:686] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.071183 4396 raft_consensus.cc:2726] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:02.071453 4394 consensus_queue.cc:237] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.073096 4258 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 1.
I20240502 21:39:02.083110 4081 catalog_manager.cc:5484] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.083921 4381 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.147156 4381 raft_consensus.cc:482] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:02.147645 4381 raft_consensus.cc:504] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.149365 4381 leader_election.cc:288] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.150272 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c00fe81035148709139976e14482b50" is_pre_election: true
I20240502 21:39:02.150525 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:02.150939 4258 raft_consensus.cc:2379] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b5a176db1134fc9a21391052977e0e9 in current term 1: Already voted for candidate bebef7198ff84fc191d1f0870cbc2237 in this term.
I20240502 21:39:02.151832 4150 leader_election.cc:302] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9; no voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:02.152516 4381 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.157524 4381 raft_consensus.cc:2726] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:02.182435 4381 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.193349 4381 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.224917 4428 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.234462 4382 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.240412 4396 raft_consensus.cc:482] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:02.240829 4396 raft_consensus.cc:504] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.242259 4396 leader_election.cc:288] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.243062 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9" is_pre_election: true
I20240502 21:39:02.243299 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237" is_pre_election: true
I20240502 21:39:02.243773 4183 raft_consensus.cc:2454] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 0.
I20240502 21:39:02.243888 4333 raft_consensus.cc:2454] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 0.
I20240502 21:39:02.244853 4224 leader_election.cc:302] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:02.245609 4396 raft_consensus.cc:2781] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:02.245934 4396 raft_consensus.cc:482] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:02.246206 4396 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.251912 4396 raft_consensus.cc:504] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.253593 4396 leader_election.cc:288] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.254261 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.254509 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.254914 4183 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.255072 4333 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:02.261678 4183 raft_consensus.cc:2454] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 1.
I20240502 21:39:02.261678 4333 raft_consensus.cc:2454] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 1.
I20240502 21:39:02.263002 4224 leader_election.cc:302] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:02.263662 4396 raft_consensus.cc:2781] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:02.264057 4396 raft_consensus.cc:686] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.264791 4396 consensus_queue.cc:237] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.271611 4080 catalog_manager.cc:5484] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.275177 4396 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.285952 4383 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.312336 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:02.312927 4258 raft_consensus.cc:595] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:02.324378 4382 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.329551 4333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c1466665aa664ba7acca06db61caf818"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:02.330101 4333 raft_consensus.cc:595] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:02.336098 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:02.336778 4258 raft_consensus.cc:595] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:02.339959 4428 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.345854 4333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:02.346490 4333 raft_consensus.cc:595] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:02.360006 4183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:56810
I20240502 21:39:02.360582 4183 raft_consensus.cc:595] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:02.365420 4396 raft_consensus.cc:979] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.365872 4396 raft_consensus.cc:1067] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.366678 4335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:02.367241 4335 raft_consensus.cc:595] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:02.367458 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56804
I20240502 21:39:02.368021 4183 raft_consensus.cc:482] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.368381 4183 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.372103 4335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:02.372615 4335 raft_consensus.cc:595] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:02.373967 4183 raft_consensus.cc:504] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.375589 4183 leader_election.cc:288] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.376571 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.376869 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b75edea9b274b8a82e195f0a5dde101" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.377154 4183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:56810
I20240502 21:39:02.377959 4335 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.377278 4259 raft_consensus.cc:3032] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.378455 4183 raft_consensus.cc:595] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:02.378407 4259 raft_consensus.cc:729] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.379498 4259 consensus_queue.cc:260] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.380789 4259 raft_consensus.cc:3037] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.382916 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:02.383389 4258 raft_consensus.cc:595] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Received request to transfer leadership to TS bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:02.385303 4335 raft_consensus.cc:2454] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.386379 4149 leader_election.cc:302] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:02.387105 4381 raft_consensus.cc:2781] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.387307 4259 raft_consensus.cc:2454] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.387488 4381 raft_consensus.cc:686] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.388329 4381 consensus_queue.cc:237] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.394342 4081 catalog_manager.cc:5484] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 1 to 2, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 2 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.439535 4428 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.449774 4382 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.457999 4456 raft_consensus.cc:979] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.458463 4456 raft_consensus.cc:1067] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.459870 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37030
I20240502 21:39:02.460364 4259 raft_consensus.cc:482] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.460803 4259 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.465528 4259 raft_consensus.cc:504] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.467381 4259 leader_election.cc:288] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.468370 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.469049 4335 raft_consensus.cc:3032] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.469147 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.469424 4335 raft_consensus.cc:729] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:02.469798 4183 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.470223 4335 consensus_queue.cc:260] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.471530 4335 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.476478 4183 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.477663 4224 leader_election.cc:302] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.477933 4335 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.478498 4396 raft_consensus.cc:2781] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.478955 4396 raft_consensus.cc:686] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.479676 4396 consensus_queue.cc:237] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.485565 4080 catalog_manager.cc:5484] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 1 to 2, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.535106 4456 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.546150 4456 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.552922 4428 raft_consensus.cc:979] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.553418 4456 raft_consensus.cc:1067] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.554827 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1466665aa664ba7acca06db61caf818"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56806
I20240502 21:39:02.555284 4183 raft_consensus.cc:482] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.555594 4183 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.560698 4183 raft_consensus.cc:504] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.562399 4183 leader_election.cc:288] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.563143 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.563421 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.563738 4259 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.563937 4335 raft_consensus.cc:3032] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.564242 4335 raft_consensus.cc:729] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:02.564952 4335 consensus_queue.cc:260] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.565948 4335 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.568745 4259 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.569658 4150 leader_election.cc:302] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.570290 4381 raft_consensus.cc:2781] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.570706 4381 raft_consensus.cc:686] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.570685 4335 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.571549 4381 consensus_queue.cc:237] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.577612 4081 catalog_manager.cc:5484] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 1 to 2, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 2 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.613241 4381 consensus_queue.cc:1035] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.623682 4381 consensus_queue.cc:1035] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.630447 4406 raft_consensus.cc:979] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.630914 4381 raft_consensus.cc:1067] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.632265 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37028
I20240502 21:39:02.632758 4259 raft_consensus.cc:482] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.633023 4259 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.638063 4259 raft_consensus.cc:504] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.640127 4259 leader_election.cc:288] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.643817 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.644469 4183 raft_consensus.cc:3032] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.644790 4183 raft_consensus.cc:729] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.645519 4183 consensus_queue.cc:260] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.646814 4183 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.649794 4394 raft_consensus.cc:979] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.650314 4404 raft_consensus.cc:1067] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.652086 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37028
I20240502 21:39:02.652760 4259 raft_consensus.cc:482] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.653108 4259 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.654125 4183 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.644793 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58c31ea9ec54aad8d051faca95b94e9" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.655174 4224 leader_election.cc:302] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.655443 4335 raft_consensus.cc:3037] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.656186 4396 raft_consensus.cc:2781] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.656708 4396 raft_consensus.cc:686] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.657531 4396 consensus_queue.cc:237] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.663460 4335 raft_consensus.cc:2454] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.664827 4081 catalog_manager.cc:5484] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 1 to 2, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.666369 4259 raft_consensus.cc:504] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.669363 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.669821 4259 leader_election.cc:288] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.669996 4183 raft_consensus.cc:3032] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.670313 4183 raft_consensus.cc:729] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.671015 4183 consensus_queue.cc:260] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.680709 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.681188 4183 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.681380 4335 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.687917 4183 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.689113 4224 leader_election.cc:302] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.689972 4335 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.690302 4383 raft_consensus.cc:2781] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.690784 4383 raft_consensus.cc:686] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.692145 4391 raft_consensus.cc:979] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.691615 4383 consensus_queue.cc:237] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.692600 4391 raft_consensus.cc:1067] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.694248 4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56804
I20240502 21:39:02.694819 4184 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.695168 4184 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.702056 4184 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.704412 4184 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.705080 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.705718 4259 raft_consensus.cc:3032] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.705080 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.706035 4259 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.706832 4259 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.708021 4259 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.708078 4080 catalog_manager.cc:5484] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 1 to 2, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.712672 4335 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.714511 4259 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.715740 4150 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.716414 4404 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.716882 4404 raft_consensus.cc:686] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.717680 4404 consensus_queue.cc:237] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.719311 4335 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.724120 4080 catalog_manager.cc:5484] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 1 to 2, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 2 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.727463 4383 consensus_queue.cc:1035] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.734717 4391 raft_consensus.cc:979] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: : Instructing follower bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:02.735183 4462 raft_consensus.cc:1067] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Signalling peer bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:02.738852 4335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:52210
I20240502 21:39:02.740080 4335 raft_consensus.cc:482] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.741118 4335 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.747263 4462 consensus_queue.cc:1035] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:02.749689 4335 raft_consensus.cc:504] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.753317 4335 leader_election.cc:288] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:02.753206 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.754312 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8e35b25514dfd91cc7c85c67147a9" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.754945 4258 raft_consensus.cc:3032] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.755259 4258 raft_consensus.cc:729] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.756016 4258 consensus_queue.cc:260] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.757114 4258 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.763113 4428 raft_consensus.cc:979] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.763645 4456 raft_consensus.cc:1067] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:02.765241 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37030
I20240502 21:39:02.766794 4258 raft_consensus.cc:2454] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 2.
I20240502 21:39:02.767416 4259 raft_consensus.cc:482] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.767971 4259 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.767974 4301 leader_election.cc:302] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters: 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:02.768965 4456 raft_consensus.cc:2781] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.769443 4456 raft_consensus.cc:686] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:02.770215 4456 consensus_queue.cc:237] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.774247 4259 raft_consensus.cc:504] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.776942 4259 leader_election.cc:288] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.776948 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:02.777582 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.777834 4184 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.777294 4081 catalog_manager.cc:5484] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 1 to 2, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 2 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:02.778244 4335 raft_consensus.cc:3032] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.778535 4335 raft_consensus.cc:729] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:02.779119 4335 consensus_queue.cc:260] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.780006 4335 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.783599 4184 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.784718 4224 leader_election.cc:302] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.785504 4462 raft_consensus.cc:2781] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.785903 4462 raft_consensus.cc:686] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:02.786651 4335 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 2.
I20240502 21:39:02.786702 4462 consensus_queue.cc:237] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.793152 4081 catalog_manager.cc:5484] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 1 to 2, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.823210 4428 raft_consensus.cc:979] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.823653 4456 raft_consensus.cc:1067] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:02.825070 4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56806
I20240502 21:39:02.825552 4184 raft_consensus.cc:482] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:02.825947 4184 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.830565 4184 raft_consensus.cc:504] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.832089 4184 leader_election.cc:288] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:02.833051 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:02.833106 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:02.833779 4259 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.833885 4335 raft_consensus.cc:3032] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:02.834224 4335 raft_consensus.cc:729] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:02.834832 4335 consensus_queue.cc:260] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.835731 4335 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:02.838487 4259 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.839514 4150 leader_election.cc:302] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:02.840147 4404 raft_consensus.cc:2781] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:02.840526 4335 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 2.
I20240502 21:39:02.840569 4404 raft_consensus.cc:686] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:02.841579 4404 consensus_queue.cc:237] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:02.847436 4081 catalog_manager.cc:5484] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 1 to 2, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 2 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:02.868965 4335 raft_consensus.cc:1261] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:02.870265 4462 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:02.877712 4184 raft_consensus.cc:1261] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:02.879225 4391 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:02.935875 4259 raft_consensus.cc:1261] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:02.937088 4404 consensus_queue.cc:1035] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:02.944447 4335 raft_consensus.cc:1261] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:02.945988 4404 consensus_queue.cc:1035] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.029691 4335 raft_consensus.cc:1261] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.030932 4404 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.038568 4259 raft_consensus.cc:1261] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.040283 4406 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.088132 4335 raft_consensus.cc:1261] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.089350 4391 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.097476 4184 raft_consensus.cc:1261] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.099408 4383 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.172325 4259 raft_consensus.cc:1261] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.173731 4456 consensus_queue.cc:1035] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.173971 4259 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.175140 4406 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.180546 4184 raft_consensus.cc:3037] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:03.185431 4335 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.187085 4406 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.189013 4184 raft_consensus.cc:1261] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.190577 4428 consensus_queue.cc:1035] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.201292 4335 raft_consensus.cc:1261] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.202775 4391 consensus_queue.cc:1035] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.210018 4183 raft_consensus.cc:1261] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.212793 4396 consensus_queue.cc:1035] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.248258 4335 raft_consensus.cc:1261] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.249493 4383 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.256691 4183 raft_consensus.cc:1261] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.258425 4396 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.283368 4335 raft_consensus.cc:1261] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.284691 4406 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:03.293012 4259 raft_consensus.cc:1261] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:03.294813 4394 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:05.412811 3527 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240502 21:39:05.413607 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:05.429659 4183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:05.430227 4183 raft_consensus.cc:595] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Received request to transfer leadership to TS bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:05.432536 4183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:05.433018 4183 raft_consensus.cc:595] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Received request to transfer leadership to TS bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:05.433935 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240502 21:39:05.434401 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:05.792539 4381 raft_consensus.cc:979] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:05.792935 4471 raft_consensus.cc:1067] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Signalling peer bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:05.794206 4335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:52226
I20240502 21:39:05.794644 4335 raft_consensus.cc:482] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:05.794895 4335 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:05.798998 4335 raft_consensus.cc:504] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:05.800292 4335 leader_election.cc:288] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 2 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:05.801306 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:05.801218 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:05.801898 4259 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:05.801980 4183 raft_consensus.cc:3032] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:05.802343 4183 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:05.803114 4183 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:05.804718 4183 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:05.808810 4259 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 2.
I20240502 21:39:05.810109 4301 leader_election.cc:302] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:05.810983 4382 raft_consensus.cc:2781] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:05.811491 4183 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 2.
I20240502 21:39:05.812054 4382 raft_consensus.cc:686] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:05.812891 4382 consensus_queue.cc:237] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:05.819666 4078 catalog_manager.cc:5484] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 1 to 2, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 2 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:05.999624 4381 raft_consensus.cc:979] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:06.000026 4471 raft_consensus.cc:1067] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Signalling peer bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:06.001359 4335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:52226
I20240502 21:39:06.001840 4335 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:06.002107 4335 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:06.007453 4335 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:06.008809 4335 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 3 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:06.009579 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:06.009781 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:06.010059 4183 raft_consensus.cc:3032] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:06.010353 4259 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:06.010407 4183 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:06.011308 4183 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:06.012377 4183 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:06.015200 4259 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 3.
I20240502 21:39:06.016651 4301 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:06.017343 4382 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:06.017839 4382 raft_consensus.cc:686] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:06.017959 4183 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 3.
I20240502 21:39:06.018697 4382 consensus_queue.cc:237] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:06.025164 4078 catalog_manager.cc:5484] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 2 to 3, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 3 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:06.200532 4259 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:06.201614 4382 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:06.211064 4183 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:06.212574 4472 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:06.435323 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:06.441128 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:06.441560 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:06.562557 4184 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:06.563809 4472 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:06.571717 4259 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:06.573012 4472 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:07.442381 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:07.447816 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:07.448244 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:08.449147 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:08.455353 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:08.455863 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:09.456833 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:09.463092 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:09.463553 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:10.464738 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:10.470791 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:10.471251 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:11.472112 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:11.478296 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:11.478719 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:12.479583 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:12.484906 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:12.485371 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:13.486184 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:13.491436 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:13.491847 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:14.492801 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:14.498346 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:14.498790 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:15.525240 3527 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240502 21:39:15.532599 4333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:15.533339 4333 raft_consensus.cc:595] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:15.541169 4333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:52240
I20240502 21:39:15.541765 4333 raft_consensus.cc:595] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:15.549273 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:15.549837 4258 raft_consensus.cc:595] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:15.554625 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:15.555114 4258 raft_consensus.cc:595] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:15.560019 4258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0"
dest_uuid: "3c00fe81035148709139976e14482b50"
mode: GRACEFUL
new_leader_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:37040
I20240502 21:39:15.560634 4258 raft_consensus.cc:595] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Received request to transfer leadership to TS 9b5a176db1134fc9a21391052977e0e9
I20240502 21:39:15.581821 4494 raft_consensus.cc:979] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.582242 4494 raft_consensus.cc:1067] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.583758 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56806
I20240502 21:39:15.584208 4183 raft_consensus.cc:482] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:15.584581 4183 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.590899 4183 raft_consensus.cc:504] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.592654 4183 leader_election.cc:288] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:15.593622 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:15.593801 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:15.594389 4258 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.594427 4333 raft_consensus.cc:3032] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:15.594831 4333 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:15.595590 4333 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.596666 4333 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.600534 4258 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.601751 4150 leader_election.cc:302] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:15.602070 4333 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.602684 4490 raft_consensus.cc:2781] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:15.604017 4490 raft_consensus.cc:686] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:15.604866 4490 consensus_queue.cc:237] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.612249 4081 catalog_manager.cc:5484] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 2 to 3, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 3 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:15.825801 4492 raft_consensus.cc:979] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.826318 4505 raft_consensus.cc:1067] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.827615 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56804
I20240502 21:39:15.828066 4183 raft_consensus.cc:482] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:15.828357 4183 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.833487 4183 raft_consensus.cc:504] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.835776 4183 leader_election.cc:288] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:15.836396 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:15.836732 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a60a625cb6242bf98c9f02d2ed4569d" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:15.837097 4258 raft_consensus.cc:3032] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:15.837287 4335 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.837498 4258 raft_consensus.cc:729] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:15.838379 4258 consensus_queue.cc:260] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.839645 4258 raft_consensus.cc:3037] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.844491 4335 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.845554 4149 leader_election.cc:302] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:15.846107 4258 raft_consensus.cc:2454] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.846242 4490 raft_consensus.cc:2781] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:15.846813 4490 raft_consensus.cc:686] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:15.847661 4490 consensus_queue.cc:237] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.854496 4081 catalog_manager.cc:5484] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 2 to 3, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 3 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:15.886727 4492 raft_consensus.cc:979] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.887135 4505 raft_consensus.cc:1067] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.888525 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56804
I20240502 21:39:15.888944 4183 raft_consensus.cc:482] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:15.889165 4183 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.893466 4183 raft_consensus.cc:504] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.894738 4183 leader_election.cc:288] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:15.895677 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:15.895674 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eeae8767c747bfa7a79c8185bc25f0" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:15.896167 4335 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.896286 4258 raft_consensus.cc:3032] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:15.896682 4258 raft_consensus.cc:729] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:15.897351 4258 consensus_queue.cc:260] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.898288 4258 raft_consensus.cc:3037] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.900900 4335 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.901831 4149 leader_election.cc:302] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:15.902542 4490 raft_consensus.cc:2781] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:15.902904 4490 raft_consensus.cc:686] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:15.903108 4258 raft_consensus.cc:2454] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:15.903748 4490 consensus_queue.cc:237] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.910637 4081 catalog_manager.cc:5484] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 2 to 3, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 3 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:15.977979 4498 raft_consensus.cc:979] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.978475 4494 raft_consensus.cc:1067] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.979816 4183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56806
I20240502 21:39:15.980237 4183 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:15.980661 4183 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:15.986138 4492 raft_consensus.cc:979] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: : Instructing follower 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.986666 4505 raft_consensus.cc:1067] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Signalling peer 9b5a176db1134fc9a21391052977e0e9 to start an election
I20240502 21:39:15.986356 4183 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.988094 4183 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 4 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:15.988233 4184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
from {username='slave'} at 127.0.0.1:56804
I20240502 21:39:15.988898 4184 raft_consensus.cc:482] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:15.988997 4258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:15.989281 4184 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.989339 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:15.989598 4258 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:15.989936 4335 raft_consensus.cc:3032] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:39:15.990239 4335 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:15.991011 4335 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.992266 4335 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:15.994509 4184 raft_consensus.cc:504] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:15.996387 4184 leader_election.cc:288] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Requested vote from peers 3c00fe81035148709139976e14482b50 (127.3.113.194:39657), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:15.997181 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:15.997752 4259 raft_consensus.cc:3032] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:15.998029 4259 raft_consensus.cc:729] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:15.998593 4335 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 4.
I20240502 21:39:15.998615 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0019d4ae8ffb4dc68e1a180f9c7e8d11" candidate_uuid: "9b5a176db1134fc9a21391052977e0e9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:15.999279 4333 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:15.998823 4259 consensus_queue.cc:260] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:16.000281 4259 raft_consensus.cc:3037] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:16.000275 4258 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 4.
I20240502 21:39:16.000388 4149 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b5a176db1134fc9a21391052977e0e9, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:16.001511 4490 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:39:16.002066 4490 raft_consensus.cc:686] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:16.002981 4490 consensus_queue.cc:237] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:16.006121 4259 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:16.006238 4333 raft_consensus.cc:2454] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b5a176db1134fc9a21391052977e0e9 in term 3.
I20240502 21:39:16.007274 4150 leader_election.cc:302] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, 9b5a176db1134fc9a21391052977e0e9; no voters:
I20240502 21:39:16.008257 4499 raft_consensus.cc:2781] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:16.009516 4499 raft_consensus.cc:686] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Becoming Leader. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:16.010556 4499 consensus_queue.cc:237] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:16.011806 4081 catalog_manager.cc:5484] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 3 to 4, leader changed from bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 4 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:16.023301 4081 catalog_manager.cc:5484] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 reported cstate change: term changed from 2 to 3, leader changed from 3c00fe81035148709139976e14482b50 (127.3.113.194) to 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193). New cstate: current_term: 3 leader_uuid: "9b5a176db1134fc9a21391052977e0e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:16.102403 4333 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.103996 4510 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.114418 4259 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.115813 4510 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.232677 4259 raft_consensus.cc:1261] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.233736 4510 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.241751 4333 raft_consensus.cc:1261] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.243022 4510 consensus_queue.cc:1035] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.353193 4259 raft_consensus.cc:1261] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.354234 4510 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.362637 4333 raft_consensus.cc:1261] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.364261 4510 consensus_queue.cc:1035] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.491158 4333 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:39:16.492473 4510 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:16.500712 4259 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:39:16.503773 4490 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:16.511456 4333 raft_consensus.cc:1261] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.512672 4499 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:16.522308 4259 raft_consensus.cc:1261] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer 9b5a176db1134fc9a21391052977e0e9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:16.523662 4510 consensus_queue.cc:1035] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:18.590957 3527 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240502 21:39:18.591601 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:18.599624 4184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:18.600129 4184 raft_consensus.cc:595] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:18.602422 4184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:18.602872 4184 raft_consensus.cc:595] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Received request to transfer leadership to TS 3c00fe81035148709139976e14482b50
I20240502 21:39:18.604764 4184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c1466665aa664ba7acca06db61caf818"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:18.605176 4184 raft_consensus.cc:595] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Received request to transfer leadership to TS bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:18.607247 4184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
mode: GRACEFUL
new_leader_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:56818
I20240502 21:39:18.607674 4184 raft_consensus.cc:595] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 LEADER]: Received request to transfer leadership to TS bebef7198ff84fc191d1f0870cbc2237
I20240502 21:39:18.608599 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240502 21:39:18.609115 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:18.654682 4527 raft_consensus.cc:979] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:18.655133 4499 raft_consensus.cc:1067] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:18.656503 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37028
I20240502 21:39:18.656952 4259 raft_consensus.cc:482] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:18.657188 4259 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:18.661655 4259 raft_consensus.cc:504] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.663426 4259 leader_election.cc:288] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 4 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:18.664287 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:18.664573 4335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f6ea4fdc1345fd908811a8ca892e32" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:18.664830 4184 raft_consensus.cc:3032] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 21:39:18.665113 4184 raft_consensus.cc:729] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:18.665146 4335 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:18.665706 4184 consensus_queue.cc:260] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.666767 4184 raft_consensus.cc:3037] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Advancing to term 4
I20240502 21:39:18.669749 4335 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 4.
I20240502 21:39:18.670790 4224 leader_election.cc:302] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:18.671481 4525 raft_consensus.cc:2781] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 21:39:18.671689 4184 raft_consensus.cc:2454] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 4.
I20240502 21:39:18.672681 4525 raft_consensus.cc:686] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 4 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:18.673491 4525 consensus_queue.cc:237] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.679914 4081 catalog_manager.cc:5484] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 3 to 4, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 4 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:18.721824 4527 raft_consensus.cc:979] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:18.722244 4499 raft_consensus.cc:1067] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 LEADER]: Signalling peer bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:18.723609 4335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:52226
I20240502 21:39:18.724027 4335 raft_consensus.cc:482] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:18.724246 4335 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:39:18.728531 4335 raft_consensus.cc:504] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.730376 4335 leader_election.cc:288] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 5 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:18.731231 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:18.731890 4184 raft_consensus.cc:3032] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 21:39:18.731732 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fd256f23a745608bc8a9cb1130675f" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:18.732179 4184 raft_consensus.cc:729] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:18.732422 4259 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:39:18.732833 4184 consensus_queue.cc:260] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.733994 4184 raft_consensus.cc:3037] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Advancing to term 5
I20240502 21:39:18.737341 4259 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 5.
I20240502 21:39:18.738343 4184 raft_consensus.cc:2454] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 5.
I20240502 21:39:18.738422 4301 leader_election.cc:302] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:18.739172 4494 raft_consensus.cc:2781] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 21:39:18.740412 4494 raft_consensus.cc:686] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 5 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:18.741191 4494 consensus_queue.cc:237] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.747565 4081 catalog_manager.cc:5484] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 4 to 5, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 5 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:18.975212 4527 raft_consensus.cc:979] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:18.975668 4527 raft_consensus.cc:1067] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Signalling peer bebef7198ff84fc191d1f0870cbc2237 to start an election
I20240502 21:39:18.977046 4333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1466665aa664ba7acca06db61caf818"
dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
from {username='slave'} at 127.0.0.1:52226
I20240502 21:39:18.977536 4333 raft_consensus.cc:482] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:18.977815 4333 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:18.982760 4333 raft_consensus.cc:504] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.984241 4333 leader_election.cc:288] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 3 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), 3c00fe81035148709139976e14482b50 (127.3.113.194:39657)
I20240502 21:39:18.985059 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:18.985145 4259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1466665aa664ba7acca06db61caf818" candidate_uuid: "bebef7198ff84fc191d1f0870cbc2237" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3c00fe81035148709139976e14482b50"
I20240502 21:39:18.985730 4184 raft_consensus.cc:3032] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:18.985754 4259 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:18.986184 4184 raft_consensus.cc:729] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:18.986946 4184 consensus_queue.cc:260] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:18.988129 4184 raft_consensus.cc:3037] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:18.992239 4259 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 3.
I20240502 21:39:18.993521 4301 leader_election.cc:302] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:18.994006 4184 raft_consensus.cc:2454] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bebef7198ff84fc191d1f0870cbc2237 in term 3.
I20240502 21:39:18.994155 4494 raft_consensus.cc:2781] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:18.995038 4494 raft_consensus.cc:686] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Becoming Leader. State: Replica: bebef7198ff84fc191d1f0870cbc2237, State: Running, Role: LEADER
I20240502 21:39:18.995787 4494 consensus_queue.cc:237] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:19.002827 4081 catalog_manager.cc:5484] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 reported cstate change: term changed from 2 to 3, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195). New cstate: current_term: 3 leader_uuid: "bebef7198ff84fc191d1f0870cbc2237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:19.101150 4499 raft_consensus.cc:979] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: : Instructing follower 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:19.101627 4499 raft_consensus.cc:1067] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Signalling peer 3c00fe81035148709139976e14482b50 to start an election
I20240502 21:39:19.103094 4259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d"
dest_uuid: "3c00fe81035148709139976e14482b50"
from {username='slave'} at 127.0.0.1:37028
I20240502 21:39:19.103513 4259 raft_consensus.cc:482] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:39:19.103811 4259 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:19.108256 4259 raft_consensus.cc:504] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:19.109573 4259 leader_election.cc:288] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 3 election: Requested vote from peers 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383), bebef7198ff84fc191d1f0870cbc2237 (127.3.113.195:37083)
I20240502 21:39:19.110365 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9b5a176db1134fc9a21391052977e0e9"
I20240502 21:39:19.110567 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ddfb10f164992a4837b435bd0d69d" candidate_uuid: "3c00fe81035148709139976e14482b50" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bebef7198ff84fc191d1f0870cbc2237"
I20240502 21:39:19.110950 4184 raft_consensus.cc:3032] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 21:39:19.111045 4333 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:19.111294 4184 raft_consensus.cc:729] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b5a176db1134fc9a21391052977e0e9, State: Running, Role: LEADER
I20240502 21:39:19.112042 4184 consensus_queue.cc:260] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:19.113333 4184 raft_consensus.cc:3037] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:19.117497 4333 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 3.
I20240502 21:39:19.118566 4224 leader_election.cc:302] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c00fe81035148709139976e14482b50, bebef7198ff84fc191d1f0870cbc2237; no voters:
I20240502 21:39:19.119112 4184 raft_consensus.cc:2454] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c00fe81035148709139976e14482b50 in term 3.
I20240502 21:39:19.119215 4525 raft_consensus.cc:2781] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:19.119752 4525 raft_consensus.cc:686] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 3 LEADER]: Becoming Leader. State: Replica: 3c00fe81035148709139976e14482b50, State: Running, Role: LEADER
I20240502 21:39:19.120601 4525 consensus_queue.cc:237] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } }
I20240502 21:39:19.125197 4184 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 5 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:39:19.126402 4494 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:39:19.127825 4081 catalog_manager.cc:5484] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 reported cstate change: term changed from 2 to 3, leader changed from 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193) to 3c00fe81035148709139976e14482b50 (127.3.113.194). New cstate: current_term: 3 leader_uuid: "3c00fe81035148709139976e14482b50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:19.139451 4259 raft_consensus.cc:1261] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 5 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240502 21:39:19.141098 4494 consensus_queue.cc:1035] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:39:19.205265 4333 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:39:19.206483 4525 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:19.215710 4184 raft_consensus.cc:1261] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240502 21:39:19.217003 4528 consensus_queue.cc:1035] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:19.510960 4184 raft_consensus.cc:1261] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:19.512200 4525 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:19.522801 4333 raft_consensus.cc:1261] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Refusing update from remote peer 3c00fe81035148709139976e14482b50: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:19.523981 4550 consensus_queue.cc:1035] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebef7198ff84fc191d1f0870cbc2237" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:19.589159 4259 raft_consensus.cc:1261] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:19.591197 4494 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c00fe81035148709139976e14482b50" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 39657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:19.600320 4184 raft_consensus.cc:1261] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Refusing update from remote peer bebef7198ff84fc191d1f0870cbc2237: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240502 21:39:19.603410 4494 consensus_queue.cc:1035] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5a176db1134fc9a21391052977e0e9" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:19.610405 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:19.623128 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:19.623647 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:20.624619 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:20.631738 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:20.632258 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:21.633082 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:21.639204 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:21.639694 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:22.640547 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:22.645752 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:22.646186 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:23.646934 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:23.652179 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:23.652668 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:24.653617 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:24.659622 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:24.660072 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:25.661026 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:25.667196 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:25.667636 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:26.668557 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:26.673849 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:26.674295 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:27.675215 3527 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240502 21:39:27.680922 3527 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240502 21:39:27.681393 3527 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240502 21:39:28.709028 3527 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240502 21:39:28.709525 3527 tablet_server.cc:178] TabletServer@127.3.113.193:0 shutting down...
I20240502 21:39:28.735854 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:39:28.736825 3527 tablet_replica.cc:330] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.737430 3527 raft_consensus.cc:2229] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.737974 3527 raft_consensus.cc:2258] T 4af8e35b25514dfd91cc7c85c67147a9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.739955 3527 tablet_replica.cc:330] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.740571 3527 raft_consensus.cc:2229] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.740975 3527 raft_consensus.cc:2258] T c1466665aa664ba7acca06db61caf818 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.742600 3527 tablet_replica.cc:330] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.743170 3527 raft_consensus.cc:2229] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.743580 3527 raft_consensus.cc:2258] T 361ddfb10f164992a4837b435bd0d69d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.745277 3527 tablet_replica.cc:330] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.745786 3527 raft_consensus.cc:2229] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.746218 3527 raft_consensus.cc:2258] T 94f6ea4fdc1345fd908811a8ca892e32 P 9b5a176db1134fc9a21391052977e0e9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.747901 3527 tablet_replica.cc:330] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.748445 3527 raft_consensus.cc:2229] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.749266 3527 raft_consensus.cc:2258] T 57eeae8767c747bfa7a79c8185bc25f0 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.750897 3527 tablet_replica.cc:330] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.751361 3527 raft_consensus.cc:2229] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.751775 3527 raft_consensus.cc:2258] T e58c31ea9ec54aad8d051faca95b94e9 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.753556 3527 tablet_replica.cc:330] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.754001 3527 raft_consensus.cc:2229] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.754760 3527 raft_consensus.cc:2258] T 5b75edea9b274b8a82e195f0a5dde101 P 9b5a176db1134fc9a21391052977e0e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.756376 3527 tablet_replica.cc:330] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.756824 3527 raft_consensus.cc:2229] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.757552 3527 raft_consensus.cc:2258] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.759155 3527 tablet_replica.cc:330] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.759595 3527 raft_consensus.cc:2229] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.759997 3527 raft_consensus.cc:2258] T d0fd256f23a745608bc8a9cb1130675f P 9b5a176db1134fc9a21391052977e0e9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.761749 3527 tablet_replica.cc:330] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9: stopping tablet replica
I20240502 21:39:28.762208 3527 raft_consensus.cc:2229] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.763064 3527 raft_consensus.cc:2258] T 5a60a625cb6242bf98c9f02d2ed4569d P 9b5a176db1134fc9a21391052977e0e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.795172 3527 tablet_server.cc:194] TabletServer@127.3.113.193:0 shutdown complete.
I20240502 21:39:28.815220 3527 tablet_server.cc:178] TabletServer@127.3.113.194:0 shutting down...
I20240502 21:39:28.841676 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:39:28.842417 3527 tablet_replica.cc:330] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.842989 3527 raft_consensus.cc:2229] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.843472 3527 raft_consensus.cc:2258] T 4af8e35b25514dfd91cc7c85c67147a9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.845309 3527 tablet_replica.cc:330] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.845827 3527 raft_consensus.cc:2229] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.846227 3527 raft_consensus.cc:2258] T c1466665aa664ba7acca06db61caf818 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.847879 3527 tablet_replica.cc:330] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.848449 3527 raft_consensus.cc:2229] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.849185 3527 raft_consensus.cc:2258] T 361ddfb10f164992a4837b435bd0d69d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.850785 3527 tablet_replica.cc:330] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.851275 3527 raft_consensus.cc:2229] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.851704 3527 raft_consensus.cc:2258] T 5a60a625cb6242bf98c9f02d2ed4569d P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.853503 3527 tablet_replica.cc:330] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.853972 3527 raft_consensus.cc:2229] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.854436 3527 raft_consensus.cc:2258] T 57eeae8767c747bfa7a79c8185bc25f0 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.856387 3527 tablet_replica.cc:330] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50: stopping tablet replica
W20240502 21:39:28.856597 4224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.113.193:35383: connect: Connection refused (error 111) [suppressed 3 similar messages]
I20240502 21:39:28.856920 3527 raft_consensus.cc:2229] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 4 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.858039 3527 raft_consensus.cc:2258] T 94f6ea4fdc1345fd908811a8ca892e32 P 3c00fe81035148709139976e14482b50 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.860195 3527 tablet_replica.cc:330] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50: stopping tablet replica
W20240502 21:39:28.860834 4224 consensus_peers.cc:485] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 -> Peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Couldn't send request to peer 9b5a176db1134fc9a21391052977e0e9. Status: Network error: Client connection negotiation failed: client connection to 127.3.113.193:35383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:28.860977 3527 raft_consensus.cc:2229] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.861485 3527 raft_consensus.cc:2258] T d0fd256f23a745608bc8a9cb1130675f P 3c00fe81035148709139976e14482b50 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.863278 3527 tablet_replica.cc:330] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.863795 3527 raft_consensus.cc:2229] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.864195 3527 raft_consensus.cc:2258] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P 3c00fe81035148709139976e14482b50 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.866096 3527 tablet_replica.cc:330] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.866554 3527 raft_consensus.cc:2229] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.867300 3527 raft_consensus.cc:2258] T e58c31ea9ec54aad8d051faca95b94e9 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.868992 3527 tablet_replica.cc:330] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50: stopping tablet replica
I20240502 21:39:28.869429 3527 raft_consensus.cc:2229] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.869824 3527 raft_consensus.cc:2258] T 5b75edea9b274b8a82e195f0a5dde101 P 3c00fe81035148709139976e14482b50 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 21:39:28.896749 4300 consensus_peers.cc:485] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 -> Peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Couldn't send request to peer 9b5a176db1134fc9a21391052977e0e9. Status: Network error: Client connection negotiation failed: client connection to 127.3.113.193:35383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:28.900555 3527 tablet_server.cc:194] TabletServer@127.3.113.194:0 shutdown complete.
W20240502 21:39:28.903698 4300 consensus_peers.cc:485] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 -> Peer 9b5a176db1134fc9a21391052977e0e9 (127.3.113.193:35383): Couldn't send request to peer 9b5a176db1134fc9a21391052977e0e9. Status: Network error: Client connection negotiation failed: client connection to 127.3.113.193:35383: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:39:28.910943 4301 consensus_peers.cc:485] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 -> Peer 3c00fe81035148709139976e14482b50 (127.3.113.194:39657): Couldn't send request to peer 3c00fe81035148709139976e14482b50. Status: Network error: Client connection negotiation failed: client connection to 127.3.113.194:39657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:28.923478 3527 tablet_server.cc:178] TabletServer@127.3.113.195:0 shutting down...
I20240502 21:39:28.956380 3527 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240502 21:39:28.957316 3527 tablet_replica.cc:330] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.958545 3527 raft_consensus.cc:2229] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.959640 3527 raft_consensus.cc:2258] T 4af8e35b25514dfd91cc7c85c67147a9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.961984 3527 tablet_replica.cc:330] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.962577 3527 raft_consensus.cc:2229] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 5 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.963341 3527 raft_consensus.cc:2258] T d0fd256f23a745608bc8a9cb1130675f P bebef7198ff84fc191d1f0870cbc2237 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.965207 3527 tablet_replica.cc:330] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.965775 3527 raft_consensus.cc:2229] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.966269 3527 raft_consensus.cc:2258] T 0019d4ae8ffb4dc68e1a180f9c7e8d11 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.968050 3527 tablet_replica.cc:330] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.968636 3527 raft_consensus.cc:2229] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.969084 3527 raft_consensus.cc:2258] T 361ddfb10f164992a4837b435bd0d69d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.971037 3527 tablet_replica.cc:330] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.971583 3527 raft_consensus.cc:2229] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 3 LEADER]: Raft consensus shutting down.
I20240502 21:39:28.972541 3527 raft_consensus.cc:2258] T c1466665aa664ba7acca06db61caf818 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.974474 3527 tablet_replica.cc:330] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.975033 3527 raft_consensus.cc:2229] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.975523 3527 raft_consensus.cc:2258] T e58c31ea9ec54aad8d051faca95b94e9 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.977481 3527 tablet_replica.cc:330] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.978041 3527 raft_consensus.cc:2229] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.978528 3527 raft_consensus.cc:2258] T 5b75edea9b274b8a82e195f0a5dde101 P bebef7198ff84fc191d1f0870cbc2237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.980475 3527 tablet_replica.cc:330] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.981004 3527 raft_consensus.cc:2229] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.981527 3527 raft_consensus.cc:2258] T 5a60a625cb6242bf98c9f02d2ed4569d P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.983538 3527 tablet_replica.cc:330] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.984079 3527 raft_consensus.cc:2229] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.984606 3527 raft_consensus.cc:2258] T 57eeae8767c747bfa7a79c8185bc25f0 P bebef7198ff84fc191d1f0870cbc2237 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:28.986481 3527 tablet_replica.cc:330] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237: stopping tablet replica
I20240502 21:39:28.986965 3527 raft_consensus.cc:2229] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:28.987392 3527 raft_consensus.cc:2258] T 94f6ea4fdc1345fd908811a8ca892e32 P bebef7198ff84fc191d1f0870cbc2237 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:29.018023 3527 tablet_server.cc:194] TabletServer@127.3.113.195:0 shutdown complete.
I20240502 21:39:29.038007 3527 master.cc:537] Master@127.3.113.254:46397 shutting down...
I20240502 21:39:29.068046 3527 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 1 LEADER]: Raft consensus shutting down.
I20240502 21:39:29.068917 3527 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:29.069322 3527 tablet_replica.cc:330] T 00000000000000000000000000000000 P aef88888bce9480fad615b0ac9103455: stopping tablet replica
I20240502 21:39:29.088951 3527 master.cc:552] Master@127.3.113.254:46397 shutdown complete.
3c00fe81035148709139976e14482b50 4
9b5a176db1134fc9a21391052977e0e9 5
bebef7198ff84fc191d1f0870cbc2237 1
3c00fe81035148709139976e14482b50 4
9b5a176db1134fc9a21391052977e0e9 3
bebef7198ff84fc191d1f0870cbc2237 3
3c00fe81035148709139976e14482b50 1
9b5a176db1134fc9a21391052977e0e9 8
bebef7198ff84fc191d1f0870cbc2237 1
3c00fe81035148709139976e14482b50 3
9b5a176db1134fc9a21391052977e0e9 4
bebef7198ff84fc191d1f0870cbc2237 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28606 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240502 21:39:29.133898 3527 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.113.254:39185
I20240502 21:39:29.135048 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:29.142645 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.142628 4570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.143497 4572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:29.143990 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:29.145457 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:29.145658 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:29.145830 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685969145802 us; error 0 us; skew 500 ppm
I20240502 21:39:29.146373 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:29.155246 3527 webserver.cc:460] Webserver started at http://127.3.113.254:38915/ using document root <none> and password file <none>
I20240502 21:39:29.155809 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:29.156049 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:29.156404 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:29.157683 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/master-0-root/instance:
uuid: "46211e964314466eab7da59d60aa017c"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.162570 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 21:39:29.166226 4577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.167102 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:39:29.167384 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/master-0-root
uuid: "46211e964314466eab7da59d60aa017c"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.167699 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:29.186321 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:29.187659 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:29.228497 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.254:39185
I20240502 21:39:29.228572 4629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.254:39185 every 8 connection(s)
I20240502 21:39:29.232524 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:29.242769 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c: Bootstrap starting.
I20240502 21:39:29.247078 4630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:29.251443 4630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c: No bootstrap required, opened a new log
I20240502 21:39:29.253475 4630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER }
I20240502 21:39:29.253960 4630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:29.254173 4630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46211e964314466eab7da59d60aa017c, State: Initialized, Role: FOLLOWER
I20240502 21:39:29.254729 4630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER }
I20240502 21:39:29.255163 4630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:39:29.255353 4630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:39:29.255582 4630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:29.259786 4630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER }
I20240502 21:39:29.260326 4630 leader_election.cc:302] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46211e964314466eab7da59d60aa017c; no voters:
I20240502 21:39:29.261561 4630 leader_election.cc:288] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:39:29.261854 4633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:29.263331 4633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [term 1 LEADER]: Becoming Leader. State: Replica: 46211e964314466eab7da59d60aa017c, State: Running, Role: LEADER
I20240502 21:39:29.264292 4633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER }
I20240502 21:39:29.264942 4630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:39:29.272560 4634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46211e964314466eab7da59d60aa017c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER } }
I20240502 21:39:29.272459 4635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46211e964314466eab7da59d60aa017c. Latest consensus state: current_term: 1 leader_uuid: "46211e964314466eab7da59d60aa017c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46211e964314466eab7da59d60aa017c" member_type: VOTER } }
I20240502 21:39:29.273193 4634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [sys.catalog]: This master's current role is: LEADER
I20240502 21:39:29.273383 4635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c [sys.catalog]: This master's current role is: LEADER
I20240502 21:39:29.275194 4641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:39:29.283283 4641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:39:29.284372 3527 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 21:39:29.292454 4641 catalog_manager.cc:1347] Generated new cluster ID: 9b648dfd9bc24dd4bc9a582d9f8af254
I20240502 21:39:29.292783 4641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:39:29.318831 4641 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:39:29.320152 4641 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:39:29.334375 4641 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 46211e964314466eab7da59d60aa017c: Generated new TSK 0
I20240502 21:39:29.335024 4641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:39:29.351836 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:29.359093 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.360767 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.363991 4656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:29.365020 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:29.366410 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:29.366689 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:29.366923 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685969366896 us; error 0 us; skew 500 ppm
I20240502 21:39:29.367816 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:29.371443 3527 webserver.cc:460] Webserver started at http://127.3.113.193:39533/ using document root <none> and password file <none>
I20240502 21:39:29.371966 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:29.372143 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:29.372475 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:29.373658 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/instance:
uuid: "09ff1ee782864ac99d05a7f011aeef44"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.378500 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:39:29.382086 4661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.382931 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:39:29.383221 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root
uuid: "09ff1ee782864ac99d05a7f011aeef44"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.383538 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:29.398667 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:29.399991 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:29.401650 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:29.404048 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:29.404254 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.404546 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:29.404727 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.452428 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.193:44335
I20240502 21:39:29.452517 4723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.193:44335 every 8 connection(s)
I20240502 21:39:29.457748 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:29.465835 4728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.466950 4729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.469674 4731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:29.472352 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:29.472640 4724 heartbeater.cc:347] Connected to a master server at 127.3.113.254:39185
I20240502 21:39:29.473090 4724 heartbeater.cc:464] Registering TS with master...
I20240502 21:39:29.474170 4724 heartbeater.cc:511] Master 127.3.113.254:39185 requested a full tablet report, sending...
I20240502 21:39:29.474395 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:29.474676 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:29.474874 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685969474853 us; error 0 us; skew 500 ppm
I20240502 21:39:29.475661 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:29.476919 4594 ts_manager.cc:194] Registered new tserver with Master: 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:29.478876 3527 webserver.cc:460] Webserver started at http://127.3.113.194:43297/ using document root <none> and password file <none>
I20240502 21:39:29.479238 4594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33028
I20240502 21:39:29.479447 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:29.479720 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:29.479996 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:29.481791 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/instance:
uuid: "ddea410051614f23b93c7342ec7cc545"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.486955 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 21:39:29.490622 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.491477 3527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:39:29.491781 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root
uuid: "ddea410051614f23b93c7342ec7cc545"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.492051 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:29.517624 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:29.519140 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:29.520741 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:29.523113 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:29.523324 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.523573 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:29.523774 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.567603 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.194:38679
I20240502 21:39:29.567725 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.194:38679 every 8 connection(s)
I20240502 21:39:29.572463 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:29.581496 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:29.582459 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:29.585386 4802 heartbeater.cc:347] Connected to a master server at 127.3.113.254:39185
I20240502 21:39:29.585615 3527 server_base.cc:1034] running on GCE node
I20240502 21:39:29.585906 4802 heartbeater.cc:464] Registering TS with master...
W20240502 21:39:29.586373 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:29.587116 4802 heartbeater.cc:511] Master 127.3.113.254:39185 requested a full tablet report, sending...
I20240502 21:39:29.587457 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:29.587708 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:29.587878 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685969587865 us; error 0 us; skew 500 ppm
I20240502 21:39:29.588481 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:29.589329 4594 ts_manager.cc:194] Registered new tserver with Master: ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:29.590873 4594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33034
I20240502 21:39:29.591368 3527 webserver.cc:460] Webserver started at http://127.3.113.195:42193/ using document root <none> and password file <none>
I20240502 21:39:29.592003 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:29.592228 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:29.592535 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:29.593812 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/instance:
uuid: "df8e5a877b6640d692917083c9fb5282"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.598505 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:39:29.624811 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.625903 3527 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20240502 21:39:29.626235 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root
uuid: "df8e5a877b6640d692917083c9fb5282"
format_stamp: "Formatted at 2024-05-02 21:39:29 on dist-test-slave-pddb"
I20240502 21:39:29.626721 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:29.648396 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:29.649870 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:29.651540 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:29.654256 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:29.654503 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.654829 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:29.654985 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:29.708562 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.195:45259
I20240502 21:39:29.708675 4876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.195:45259 every 8 connection(s)
I20240502 21:39:29.725766 4877 heartbeater.cc:347] Connected to a master server at 127.3.113.254:39185
I20240502 21:39:29.726141 4877 heartbeater.cc:464] Registering TS with master...
I20240502 21:39:29.726938 4877 heartbeater.cc:511] Master 127.3.113.254:39185 requested a full tablet report, sending...
I20240502 21:39:29.729770 4594 ts_manager.cc:194] Registered new tserver with Master: df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:29.730683 3527 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018031949s
I20240502 21:39:29.731940 4594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33046
I20240502 21:39:29.734304 3527 test_util.cc:268] Using random seed: 1881184906
I20240502 21:39:29.761226 4594 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33058:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240502 21:39:29.768072 4594 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:39:29.979368 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8321179b7548c08b34565f04746387 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 21:39:29.979846 4841 tablet_service.cc:1467] Processing CreateTablet for tablet f4d1e2fc15594054a6cd01aab641f8cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 21:39:29.979472 4842 tablet_service.cc:1467] Processing CreateTablet for tablet fad5baefdf3c477b9ac99551bd3263ae (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 21:39:29.981361 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d1e2fc15594054a6cd01aab641f8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:29.982065 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5baefdf3c477b9ac99551bd3263ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:29.983814 4840 tablet_service.cc:1467] Processing CreateTablet for tablet 21a62e8a8c024197ab0afb8c0c284829 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 21:39:29.984659 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8321179b7548c08b34565f04746387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:29.985376 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a62e8a8c024197ab0afb8c0c284829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:29.990680 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 047156e0297249998c3f2c5362c70432 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 21:39:29.995412 4837 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5e2e41273949e2afa8bc07812c9ca7 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 21:39:29.998997 4836 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 21:39:30.002769 4835 tablet_service.cc:1467] Processing CreateTablet for tablet b26fb0cda47f46d0b3fa153864652df0 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 21:39:30.006803 4834 tablet_service.cc:1467] Processing CreateTablet for tablet ba7a6cb1d5034911923934eee0d68132 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 21:39:30.012073 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047156e0297249998c3f2c5362c70432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.013041 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5e2e41273949e2afa8bc07812c9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.013810 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.015784 4840 tablet_service.cc:1467] Processing CreateTablet for tablet 856dc0c130c04f4894884164ae24a053 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 21:39:30.016230 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 21:39:30.017961 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856dc0c130c04f4894884164ae24a053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.020212 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.023403 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7a6cb1d5034911923934eee0d68132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.024981 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26fb0cda47f46d0b3fa153864652df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.042743 4689 tablet_service.cc:1467] Processing CreateTablet for tablet fad5baefdf3c477b9ac99551bd3263ae (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 21:39:30.044425 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5baefdf3c477b9ac99551bd3263ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.050100 4688 tablet_service.cc:1467] Processing CreateTablet for tablet f4d1e2fc15594054a6cd01aab641f8cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 21:39:30.051095 4840 tablet_service.cc:1467] Processing CreateTablet for tablet e904cd5cd3b848d3b8dd89329851ba37 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 21:39:30.051548 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d1e2fc15594054a6cd01aab641f8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.052474 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e904cd5cd3b848d3b8dd89329851ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.058058 4687 tablet_service.cc:1467] Processing CreateTablet for tablet 21a62e8a8c024197ab0afb8c0c284829 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 21:39:30.059509 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a62e8a8c024197ab0afb8c0c284829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.086361 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8321179b7548c08b34565f04746387 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 21:39:30.087895 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8321179b7548c08b34565f04746387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.099484 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 07a57b1b34a74513b44d19508bf21459 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 21:39:30.106139 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 047156e0297249998c3f2c5362c70432 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 21:39:30.109067 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047156e0297249998c3f2c5362c70432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.109670 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5e2e41273949e2afa8bc07812c9ca7 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 21:39:30.111739 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5e2e41273949e2afa8bc07812c9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.121527 4836 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0c2f1f80b4015b25c866ee3d0c845 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 21:39:30.123229 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a57b1b34a74513b44d19508bf21459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.139930 4766 tablet_service.cc:1467] Processing CreateTablet for tablet fad5baefdf3c477b9ac99551bd3263ae (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240502 21:39:30.145679 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad5baefdf3c477b9ac99551bd3263ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.145279 4765 tablet_service.cc:1467] Processing CreateTablet for tablet f4d1e2fc15594054a6cd01aab641f8cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240502 21:39:30.147239 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d1e2fc15594054a6cd01aab641f8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.159178 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0c2f1f80b4015b25c866ee3d0c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.140188 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 73bd5217e94b49bab1732f9fd0c8d79c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 21:39:30.165192 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73bd5217e94b49bab1732f9fd0c8d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.169421 4840 tablet_service.cc:1467] Processing CreateTablet for tablet 515491e619e145a292c5e5f654cc63ee (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 21:39:30.170274 4764 tablet_service.cc:1467] Processing CreateTablet for tablet 21a62e8a8c024197ab0afb8c0c284829 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240502 21:39:30.170841 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515491e619e145a292c5e5f654cc63ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.171741 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a62e8a8c024197ab0afb8c0c284829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.173272 4842 tablet_service.cc:1467] Processing CreateTablet for tablet 21fbea6bad03441f8109d77239412009 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 21:39:30.174623 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fbea6bad03441f8109d77239412009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.182761 4896 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:30.183027 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 21:39:30.184739 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.185350 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8321179b7548c08b34565f04746387 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240502 21:39:30.190558 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a8321179b7548c08b34565f04746387. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.191144 4895 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:30.197301 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 047156e0297249998c3f2c5362c70432 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240502 21:39:30.198827 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047156e0297249998c3f2c5362c70432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.202234 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5e2e41273949e2afa8bc07812c9ca7 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240502 21:39:30.203768 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5e2e41273949e2afa8bc07812c9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.204808 4835 tablet_service.cc:1467] Processing CreateTablet for tablet e6e03d7dc5734e2e9e52437b5a1bc124 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 21:39:30.206183 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240502 21:39:30.206492 4759 tablet_service.cc:1467] Processing CreateTablet for tablet b26fb0cda47f46d0b3fa153864652df0 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 21:39:30.207567 4687 tablet_service.cc:1467] Processing CreateTablet for tablet b26fb0cda47f46d0b3fa153864652df0 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240502 21:39:30.208098 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf3772d51c64deaa4bc7c0d8e8e6c9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.208937 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26fb0cda47f46d0b3fa153864652df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.209311 4758 tablet_service.cc:1467] Processing CreateTablet for tablet ba7a6cb1d5034911923934eee0d68132 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 21:39:30.210656 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7a6cb1d5034911923934eee0d68132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.211859 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26fb0cda47f46d0b3fa153864652df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.212841 4895 tablet_bootstrap.cc:654] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.218142 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 856dc0c130c04f4894884164ae24a053 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 21:39:30.218654 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 856dc0c130c04f4894884164ae24a053 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240502 21:39:30.219588 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856dc0c130c04f4894884164ae24a053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.220081 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856dc0c130c04f4894884164ae24a053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.221571 4682 tablet_service.cc:1467] Processing CreateTablet for tablet ba7a6cb1d5034911923934eee0d68132 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240502 21:39:30.222904 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7a6cb1d5034911923934eee0d68132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.225529 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 21:39:30.206228 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e03d7dc5734e2e9e52437b5a1bc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.227633 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.239338 4689 tablet_service.cc:1467] Processing CreateTablet for tablet e904cd5cd3b848d3b8dd89329851ba37 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 21:39:30.242169 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e904cd5cd3b848d3b8dd89329851ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.249611 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 07a57b1b34a74513b44d19508bf21459 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 21:39:30.270478 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a57b1b34a74513b44d19508bf21459. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:39:30.284943 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:30.287755 4837 tablet_service.cc:1467] Processing CreateTablet for tablet dfe02980826846ccaa247803f60d8f0c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 21:39:30.313931 4841 tablet_service.cc:1467] Processing CreateTablet for tablet 30ff5dbe7549485e96acef1baa80cd02 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 21:39:30.320680 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ff5dbe7549485e96acef1baa80cd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.326508 4895 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:30.327059 4895 ts_tablet_manager.cc:1367] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.136s user 0.043s sys 0.037s
I20240502 21:39:30.330088 4895 raft_consensus.cc:348] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.331203 4895 raft_consensus.cc:374] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:30.331526 4895 raft_consensus.cc:729] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:30.332319 4895 consensus_queue.cc:260] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.345223 4896 tablet_bootstrap.cc:654] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.308079 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 63db76a96b4c474994164c8a7086bbbd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 21:39:30.346110 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe02980826846ccaa247803f60d8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.346874 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63db76a96b4c474994164c8a7086bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.327528 4836 tablet_service.cc:1467] Processing CreateTablet for tablet 263af6fed17b44af8fc338d6aa060bff (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 21:39:30.353891 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263af6fed17b44af8fc338d6aa060bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.382710 4877 heartbeater.cc:503] Master 127.3.113.254:39185 was elected leader, sending a full tablet report...
I20240502 21:39:30.389447 4899 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:30.391170 4842 tablet_service.cc:1467] Processing CreateTablet for tablet f05bc1efec5f4b95a68f7b46fdd719ad (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 21:39:30.394068 4764 tablet_service.cc:1467] Processing CreateTablet for tablet 07a57b1b34a74513b44d19508bf21459 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240502 21:39:30.395495 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a57b1b34a74513b44d19508bf21459. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.398528 4895 ts_tablet_manager.cc:1398] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.071s user 0.012s sys 0.005s
I20240502 21:39:30.408788 4687 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0c2f1f80b4015b25c866ee3d0c845 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 21:39:30.411175 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 515491e619e145a292c5e5f654cc63ee (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 21:39:30.392218 4766 tablet_service.cc:1467] Processing CreateTablet for tablet e904cd5cd3b848d3b8dd89329851ba37 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240502 21:39:30.415093 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e904cd5cd3b848d3b8dd89329851ba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.419472 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 515491e619e145a292c5e5f654cc63ee (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240502 21:39:30.420192 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0c2f1f80b4015b25c866ee3d0c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.421561 4895 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:30.427212 4899 tablet_bootstrap.cc:654] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.427876 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05bc1efec5f4b95a68f7b46fdd719ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.407884 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 73bd5217e94b49bab1732f9fd0c8d79c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 21:39:30.430480 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73bd5217e94b49bab1732f9fd0c8d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.432662 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515491e619e145a292c5e5f654cc63ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.398609 4763 tablet_service.cc:1467] Processing CreateTablet for tablet a4d0c2f1f80b4015b25c866ee3d0c845 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240502 21:39:30.437716 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4d0c2f1f80b4015b25c866ee3d0c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.439805 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 21fbea6bad03441f8109d77239412009 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 21:39:30.391330 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240502 21:39:30.441605 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fbea6bad03441f8109d77239412009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.441893 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 071379d3b0d24e4e9cfcb3f4cb2670fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.407884 4688 tablet_service.cc:1467] Processing CreateTablet for tablet 73bd5217e94b49bab1732f9fd0c8d79c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240502 21:39:30.443289 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73bd5217e94b49bab1732f9fd0c8d79c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.446826 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515491e619e145a292c5e5f654cc63ee. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:39:30.472883 4879 tablet.cc:2369] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:39:30.483302 4724 heartbeater.cc:503] Master 127.3.113.254:39185 was elected leader, sending a full tablet report...
I20240502 21:39:30.491611 4841 tablet_service.cc:1467] Processing CreateTablet for tablet 06992601e8624b078b7f76f1d6495d96 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 21:39:30.489768 4895 tablet_bootstrap.cc:654] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.500257 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06992601e8624b078b7f76f1d6495d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.520428 4896 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:30.521090 4896 ts_tablet_manager.cc:1367] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.339s user 0.016s sys 0.004s
I20240502 21:39:30.525712 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9ceac69f5c42ed858143d4d8f7cce1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 21:39:30.527215 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9ceac69f5c42ed858143d4d8f7cce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.528627 4681 tablet_service.cc:1467] Processing CreateTablet for tablet e6e03d7dc5734e2e9e52437b5a1bc124 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 21:39:30.529951 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e03d7dc5734e2e9e52437b5a1bc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.529877 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 30ff5dbe7549485e96acef1baa80cd02 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 21:39:30.531343 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ff5dbe7549485e96acef1baa80cd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.531071 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 63db76a96b4c474994164c8a7086bbbd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 21:39:30.532052 4685 tablet_service.cc:1467] Processing CreateTablet for tablet dfe02980826846ccaa247803f60d8f0c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 21:39:30.535545 4834 tablet_service.cc:1467] Processing CreateTablet for tablet 467ec4dc8d27420e9ea3fa4584153e65 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 21:39:30.535384 4836 tablet_service.cc:1467] Processing CreateTablet for tablet 80c35cadfff54236bbf05e32f945728f (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 21:39:30.537037 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467ec4dc8d27420e9ea3fa4584153e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.537778 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c35cadfff54236bbf05e32f945728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.540810 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 21fbea6bad03441f8109d77239412009 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240502 21:39:30.548345 4759 tablet_service.cc:1467] Processing CreateTablet for tablet e6e03d7dc5734e2e9e52437b5a1bc124 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240502 21:39:30.549803 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e03d7dc5734e2e9e52437b5a1bc124. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.550572 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 30ff5dbe7549485e96acef1baa80cd02 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240502 21:39:30.551920 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ff5dbe7549485e96acef1baa80cd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.527982 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 79212407f8774e83a441fc94e7244824 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 21:39:30.558538 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79212407f8774e83a441fc94e7244824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.558415 4764 tablet_service.cc:1467] Processing CreateTablet for tablet dfe02980826846ccaa247803f60d8f0c (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240502 21:39:30.561446 4689 tablet_service.cc:1467] Processing CreateTablet for tablet 263af6fed17b44af8fc338d6aa060bff (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 21:39:30.575179 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 302a987ffb0842c0bfae54675c665763 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 21:39:30.577600 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302a987ffb0842c0bfae54675c665763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.577957 4899 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:30.578512 4899 ts_tablet_manager.cc:1367] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.189s user 0.017s sys 0.008s
I20240502 21:39:30.583796 4899 raft_consensus.cc:348] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.584877 4899 raft_consensus.cc:374] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:30.585005 4895 tablet_bootstrap.cc:492] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:30.585299 4899 raft_consensus.cc:729] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:30.585597 4895 ts_tablet_manager.cc:1367] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.164s user 0.002s sys 0.015s
I20240502 21:39:30.586241 4899 consensus_queue.cc:260] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.591351 4840 tablet_service.cc:1467] Processing CreateTablet for tablet c82b5ab84d5c47fd97758728b0244499 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 21:39:30.524178 4896 raft_consensus.cc:348] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.593772 4802 heartbeater.cc:503] Master 127.3.113.254:39185 was elected leader, sending a full tablet report...
I20240502 21:39:30.593858 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 238a1f48ddb14387bc2aa4419ff8879e (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 21:39:30.595043 4837 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3ab558007a45a7b1e07e09a6efc408 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 21:39:30.606134 4896 raft_consensus.cc:374] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:30.606617 4896 raft_consensus.cc:729] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:30.595345 4680 tablet_service.cc:1467] Processing CreateTablet for tablet f05bc1efec5f4b95a68f7b46fdd719ad (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 21:39:30.595747 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63db76a96b4c474994164c8a7086bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.610409 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 63db76a96b4c474994164c8a7086bbbd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240502 21:39:30.620724 4895 raft_consensus.cc:348] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.621183 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05bc1efec5f4b95a68f7b46fdd719ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.611433 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3ab558007a45a7b1e07e09a6efc408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.625001 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238a1f48ddb14387bc2aa4419ff8879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.617308 4898 raft_consensus.cc:482] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:30.625555 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263af6fed17b44af8fc338d6aa060bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.625535 4898 raft_consensus.cc:504] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.627846 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 06992601e8624b078b7f76f1d6495d96 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 21:39:30.629163 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe02980826846ccaa247803f60d8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.621649 4895 raft_consensus.cc:374] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:30.608474 4896 consensus_queue.cc:260] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.647631 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06992601e8624b078b7f76f1d6495d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.639156 4895 raft_consensus.cc:729] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:30.653223 4895 consensus_queue.cc:260] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.606149 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe02980826846ccaa247803f60d8f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.670365 4763 tablet_service.cc:1467] Processing CreateTablet for tablet f05bc1efec5f4b95a68f7b46fdd719ad (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240502 21:39:30.636854 4836 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71664f51b64436906f9c21a6b46d89 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 21:39:30.638981 4898 leader_election.cc:288] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:30.680028 4842 tablet_service.cc:1467] Processing CreateTablet for tablet 9166e2d9377e4adb8fdbb58e8641e2a1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 21:39:30.685545 4687 tablet_service.cc:1467] Processing CreateTablet for tablet 79212407f8774e83a441fc94e7244824 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 21:39:30.687134 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79212407f8774e83a441fc94e7244824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.688906 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9ceac69f5c42ed858143d4d8f7cce1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 21:39:30.690990 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9ceac69f5c42ed858143d4d8f7cce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.698297 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 80c35cadfff54236bbf05e32f945728f (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 21:39:30.699883 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c35cadfff54236bbf05e32f945728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.714434 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82b5ab84d5c47fd97758728b0244499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.727640 4841 tablet_service.cc:1467] Processing CreateTablet for tablet bcdd2a6ae1204bfdb7814087ba453fed (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 21:39:30.732201 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21fbea6bad03441f8109d77239412009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.758157 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 06992601e8624b078b7f76f1d6495d96 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240502 21:39:30.759647 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06992601e8624b078b7f76f1d6495d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.761883 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05bc1efec5f4b95a68f7b46fdd719ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.761596 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab81ee4feb347d38912f2d2cee03fcd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 21:39:30.765877 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63db76a96b4c474994164c8a7086bbbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.770617 4899 ts_tablet_manager.cc:1398] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.192s user 0.019s sys 0.030s
I20240502 21:39:30.774920 4899 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:30.775883 4895 ts_tablet_manager.cc:1398] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.190s user 0.003s sys 0.018s
I20240502 21:39:30.778609 4839 tablet_service.cc:1467] Processing CreateTablet for tablet e6171c5f680040b9a3efc1f98c456ea9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 21:39:30.780990 4899 tablet_bootstrap.cc:654] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.690815 4688 tablet_service.cc:1467] Processing CreateTablet for tablet 467ec4dc8d27420e9ea3fa4584153e65 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 21:39:30.788255 4766 tablet_service.cc:1467] Processing CreateTablet for tablet 9f9ceac69f5c42ed858143d4d8f7cce1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240502 21:39:30.789613 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f9ceac69f5c42ed858143d4d8f7cce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.796742 4895 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:30.800521 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:30.802457 4817 leader_election.cc:341] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:30.784061 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9166e2d9377e4adb8fdbb58e8641e2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.808476 4909 raft_consensus.cc:482] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:30.813024 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71664f51b64436906f9c21a6b46d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.789081 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 79212407f8774e83a441fc94e7244824 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240502 21:39:30.816167 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79212407f8774e83a441fc94e7244824. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.816625 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6171c5f680040b9a3efc1f98c456ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.811882 4838 tablet_service.cc:1467] Processing CreateTablet for tablet 142da03edd3d4648aae37b78118070b5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 21:39:30.819427 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142da03edd3d4648aae37b78118070b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.819478 4908 raft_consensus.cc:482] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:30.820084 4908 raft_consensus.cc:504] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.822698 4908 leader_election.cc:288] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:30.783830 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 263af6fed17b44af8fc338d6aa060bff (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240502 21:39:30.823374 4909 raft_consensus.cc:504] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.824717 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 302a987ffb0842c0bfae54675c665763 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 21:39:30.826514 4909 leader_election.cc:288] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:30.828848 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 263af6fed17b44af8fc338d6aa060bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.844715 4764 tablet_service.cc:1467] Processing CreateTablet for tablet 467ec4dc8d27420e9ea3fa4584153e65 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240502 21:39:30.842885 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdd2a6ae1204bfdb7814087ba453fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.857950 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467ec4dc8d27420e9ea3fa4584153e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.858966 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3ab558007a45a7b1e07e09a6efc408 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 21:39:30.862385 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3ab558007a45a7b1e07e09a6efc408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.866017 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:30.866765 4776 raft_consensus.cc:2454] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:30.868764 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab81ee4feb347d38912f2d2cee03fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.869370 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302a987ffb0842c0bfae54675c665763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.869045 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 302a987ffb0842c0bfae54675c665763 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240502 21:39:30.871896 4681 tablet_service.cc:1467] Processing CreateTablet for tablet c82b5ab84d5c47fd97758728b0244499 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 21:39:30.873242 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82b5ab84d5c47fd97758728b0244499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.823365 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 7702c7f8653d498685b0f0e158594b3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 21:39:30.877321 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7702c7f8653d498685b0f0e158594b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.881129 4896 ts_tablet_manager.cc:1398] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.360s user 0.010s sys 0.043s
I20240502 21:39:30.882292 4896 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:30.891839 4895 tablet_bootstrap.cc:654] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.899819 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71664f51b64436906f9c21a6b46d89 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 21:39:30.849136 4759 tablet_service.cc:1467] Processing CreateTablet for tablet 80c35cadfff54236bbf05e32f945728f (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240502 21:39:30.898334 4763 tablet_service.cc:1467] Processing CreateTablet for tablet c82b5ab84d5c47fd97758728b0244499 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240502 21:39:30.913583 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 238a1f48ddb14387bc2aa4419ff8879e (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 21:39:30.917619 4689 tablet_service.cc:1467] Processing CreateTablet for tablet bcdd2a6ae1204bfdb7814087ba453fed (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 21:39:30.941579 4837 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd1bcf32e7444ed80e0f7ae05786dde (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 21:39:30.942883 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd1bcf32e7444ed80e0f7ae05786dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.923087 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 96d93565463749aab6550e43871f7730 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 21:39:30.944134 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d93565463749aab6550e43871f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.923614 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05551863b345f78f90e88220cbb163 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 21:39:30.929279 4834 tablet_service.cc:1467] Processing CreateTablet for tablet 64a461c860474b8b9480a44b6f96c2d2 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 21:39:30.955875 4896 tablet_bootstrap.cc:654] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:30.956912 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a461c860474b8b9480a44b6f96c2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.960953 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71664f51b64436906f9c21a6b46d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.964483 4684 tablet_service.cc:1467] Processing CreateTablet for tablet e6171c5f680040b9a3efc1f98c456ea9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 21:39:30.965793 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6171c5f680040b9a3efc1f98c456ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.966497 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05551863b345f78f90e88220cbb163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.938124 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467ec4dc8d27420e9ea3fa4584153e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.986742 4899 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:30.987358 4899 ts_tablet_manager.cc:1367] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.213s user 0.043s sys 0.022s
I20240502 21:39:30.987881 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 238a1f48ddb14387bc2aa4419ff8879e (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240502 21:39:30.990518 4899 raft_consensus.cc:348] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.991325 4899 raft_consensus.cc:374] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:30.991717 4899 raft_consensus.cc:729] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:30.986865 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab81ee4feb347d38912f2d2cee03fcd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 21:39:30.993655 4899 consensus_queue.cc:260] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:30.996845 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c82b5ab84d5c47fd97758728b0244499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.997447 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 1a3ab558007a45a7b1e07e09a6efc408 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240502 21:39:30.998770 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a3ab558007a45a7b1e07e09a6efc408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:30.997752 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 142da03edd3d4648aae37b78118070b5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 21:39:31.001531 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 9e71664f51b64436906f9c21a6b46d89 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240502 21:39:31.002851 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e71664f51b64436906f9c21a6b46d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.004367 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdd2a6ae1204bfdb7814087ba453fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.006016 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c35cadfff54236bbf05e32f945728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.006562 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab81ee4feb347d38912f2d2cee03fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.008605 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302a987ffb0842c0bfae54675c665763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.009155 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142da03edd3d4648aae37b78118070b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.010193 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238a1f48ddb14387bc2aa4419ff8879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.011849 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238a1f48ddb14387bc2aa4419ff8879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.039307 4842 tablet_service.cc:1467] Processing CreateTablet for tablet c52402c2811e45eb91e35a60c0fb0701 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 21:39:31.040818 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52402c2811e45eb91e35a60c0fb0701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.045399 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 7702c7f8653d498685b0f0e158594b3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 21:39:31.051123 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7702c7f8653d498685b0f0e158594b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.051424 4762 tablet_service.cc:1467] Processing CreateTablet for tablet bcdd2a6ae1204bfdb7814087ba453fed (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240502 21:39:31.053058 4841 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 21:39:31.054644 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.065521 4817 leader_election.cc:302] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:31.048734 4836 tablet_service.cc:1467] Processing CreateTablet for tablet f0fda9381a074590aec89148d7b517cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 21:39:31.068117 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 9166e2d9377e4adb8fdbb58e8641e2a1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 21:39:31.069041 4687 tablet_service.cc:1467] Processing CreateTablet for tablet 64a461c860474b8b9480a44b6f96c2d2 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 21:39:31.072700 4837 tablet_service.cc:1467] Processing CreateTablet for tablet a1a911c9636f426e9d08bee71f60eaab (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 21:39:31.073318 4766 tablet_service.cc:1467] Processing CreateTablet for tablet 9166e2d9377e4adb8fdbb58e8641e2a1 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240502 21:39:31.080704 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a461c860474b8b9480a44b6f96c2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.074568 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcdd2a6ae1204bfdb7814087ba453fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.080088 4908 raft_consensus.cc:2781] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.082871 4908 raft_consensus.cc:482] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.083235 4908 raft_consensus.cc:3037] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.075378 4688 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd1bcf32e7444ed80e0f7ae05786dde (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 21:39:31.085974 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9166e2d9377e4adb8fdbb58e8641e2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.086038 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:31.078459 4817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 101310us
I20240502 21:39:31.090901 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd1bcf32e7444ed80e0f7ae05786dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.101969 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9166e2d9377e4adb8fdbb58e8641e2a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.107915 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab81ee4feb347d38912f2d2cee03fcd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240502 21:39:31.109376 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab81ee4feb347d38912f2d2cee03fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.112794 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.113147 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a911c9636f426e9d08bee71f60eaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.113909 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fda9381a074590aec89148d7b517cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:39:31.121332 4818 leader_election.cc:341] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.121438 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.080726 4680 tablet_service.cc:1467] Processing CreateTablet for tablet 96d93565463749aab6550e43871f7730 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 21:39:31.121985 4818 leader_election.cc:302] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:31.123042 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d93565463749aab6550e43871f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.123421 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.122290 4699 raft_consensus.cc:2454] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:31.124114 4776 raft_consensus.cc:2454] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:31.124956 4898 raft_consensus.cc:2726] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.123826 4835 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb1726ee5764d42806fc4c077226bf9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 21:39:31.126291 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb1726ee5764d42806fc4c077226bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.129062 4664 leader_election.cc:302] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:31.130143 4910 raft_consensus.cc:2781] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.130535 4910 raft_consensus.cc:482] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.130875 4910 raft_consensus.cc:3037] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.125948 4838 tablet_service.cc:1467] Processing CreateTablet for tablet ed2181a7e8194705934dadd2d4303b8a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 21:39:31.134102 4895 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:31.138461 4759 tablet_service.cc:1467] Processing CreateTablet for tablet 142da03edd3d4648aae37b78118070b5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240502 21:39:31.138705 4910 raft_consensus.cc:504] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.139551 4899 ts_tablet_manager.cc:1398] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.152s user 0.004s sys 0.006s
I20240502 21:39:31.139196 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05551863b345f78f90e88220cbb163 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 21:39:31.131419 4764 tablet_service.cc:1467] Processing CreateTablet for tablet e6171c5f680040b9a3efc1f98c456ea9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240502 21:39:31.154692 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05551863b345f78f90e88220cbb163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.155472 4910 leader_election.cc:288] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.140846 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 7702c7f8653d498685b0f0e158594b3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240502 21:39:31.145682 4895 ts_tablet_manager.cc:1367] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.349s user 0.023s sys 0.007s
I20240502 21:39:31.165565 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:31.145906 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 91e028b61d3e4a92b66697049b88e05a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 21:39:31.166350 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2181a7e8194705934dadd2d4303b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.167084 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e028b61d3e4a92b66697049b88e05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.167690 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:31.168278 4776 raft_consensus.cc:3037] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.170833 4840 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 29) took 1088 ms. Trace:
I20240502 21:39:31.174149 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142da03edd3d4648aae37b78118070b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.174257 4689 tablet_service.cc:1467] Processing CreateTablet for tablet f0fda9381a074590aec89148d7b517cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 21:39:31.131258 4908 raft_consensus.cc:504] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.175570 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fda9381a074590aec89148d7b517cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.177064 4852 raft_consensus.cc:2379] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:31.178329 4896 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.180902 4908 leader_election.cc:288] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.182912 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:31.183807 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7702c7f8653d498685b0f0e158594b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.184487 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a62e8a8c024197ab0afb8c0c284829" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:31.185313 4699 raft_consensus.cc:2379] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:31.185451 4896 ts_tablet_manager.cc:1367] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.303s user 0.010s sys 0.015s
I20240502 21:39:31.186326 4895 raft_consensus.cc:348] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.187112 4895 raft_consensus.cc:374] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.187417 4895 raft_consensus.cc:729] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.187119 4818 leader_election.cc:302] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:31.188663 4908 raft_consensus.cc:2726] T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.189442 4834 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65cfc531344398b70ce5862b2211fd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 21:39:31.190748 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c65cfc531344398b70ce5862b2211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.191056 4776 raft_consensus.cc:2454] T 21a62e8a8c024197ab0afb8c0c284829 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:31.192449 4899 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.192400 4664 leader_election.cc:302] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.192498 4682 tablet_service.cc:1467] Processing CreateTablet for tablet c52402c2811e45eb91e35a60c0fb0701 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 21:39:31.188230 4895 consensus_queue.cc:260] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.194185 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6171c5f680040b9a3efc1f98c456ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.195449 4895 ts_tablet_manager.cc:1398] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.031s user 0.001s sys 0.005s
I20240502 21:39:31.196511 4910 raft_consensus.cc:2781] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:31.197646 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd1bcf32e7444ed80e0f7ae05786dde (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240502 21:39:31.198410 4910 raft_consensus.cc:686] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:31.198908 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd1bcf32e7444ed80e0f7ae05786dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.199404 4910 consensus_queue.cc:237] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.196059 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 64a461c860474b8b9480a44b6f96c2d2 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240502 21:39:31.202225 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a461c860474b8b9480a44b6f96c2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.203220 4899 tablet_bootstrap.cc:654] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.205731 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 96d93565463749aab6550e43871f7730 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240502 21:39:31.206985 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d93565463749aab6550e43871f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.213464 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 21:39:31.213809 4841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 45) took 1013 ms. Trace:
I20240502 21:39:31.214413 4841 rpcz_store.cc:276] 0502 21:39:30.200627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.051921 (+851294us) service_pool.cc:225] Handling call
0502 21:39:31.054104 (+ 2183us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.054231 (+ 127us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.095824 (+ 41593us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.213281 (+117457us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.213742 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.215879 4896 raft_consensus.cc:348] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.216751 4896 raft_consensus.cc:374] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.217057 4896 raft_consensus.cc:729] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.217772 4896 consensus_queue.cc:260] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.218829 4895 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:31.219869 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52402c2811e45eb91e35a60c0fb0701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.220017 4896 ts_tablet_manager.cc:1398] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20240502 21:39:31.222191 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.231971 4833 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 41) took 1037 ms. Trace:
I20240502 21:39:31.228273 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 9e05551863b345f78f90e88220cbb163 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240502 21:39:31.232452 4833 rpcz_store.cc:276] 0502 21:39:30.199130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:30.921010 (+721880us) service_pool.cc:225] Handling call
0502 21:39:30.943537 (+ 22527us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:30.943683 (+ 146us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.113639 (+169956us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.231465 (+117826us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.231908 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.233528 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e05551863b345f78f90e88220cbb163. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.234464 4833 tablet_service.cc:1467] Processing CreateTablet for tablet f14af5e6ab984917b771589cdb45a779 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 21:39:31.171243 4840 rpcz_store.cc:276] 0502 21:39:30.082645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:30.590349 (+507704us) service_pool.cc:225] Handling call
0502 21:39:30.646024 (+ 55675us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:30.646180 (+ 156us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:30.978351 (+332171us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.170342 (+191991us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.170770 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.236996 4840 tablet_service.cc:1467] Processing CreateTablet for tablet d296f3d569f143fca949d49feb27d19d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 21:39:31.245957 4766 tablet_service.cc:1467] Processing CreateTablet for tablet c52402c2811e45eb91e35a60c0fb0701 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240502 21:39:31.247803 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c52402c2811e45eb91e35a60c0fb0701. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.217396 4841 tablet_service.cc:1467] Processing CreateTablet for tablet a06147be82454bb0a23d0f3e68c06f04 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 21:39:31.266048 4896 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:31.268620 4684 tablet_service.cc:1467] Processing CreateTablet for tablet a1a911c9636f426e9d08bee71f60eaab (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
W20240502 21:39:31.288463 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:31.288548 4842 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 43) took 1088 ms. Trace:
I20240502 21:39:31.289007 4842 rpcz_store.cc:276] 0502 21:39:30.199897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.038315 (+838418us) service_pool.cc:225] Handling call
0502 21:39:31.040272 (+ 1957us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.040410 (+ 138us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.076975 (+ 36565us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.241763 (+164788us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.288481 (+ 46718us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.293484 4899 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.294124 4899 ts_tablet_manager.cc:1367] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.001s
I20240502 21:39:31.296850 4766 tablet_service.cc:1467] Processing CreateTablet for tablet f0fda9381a074590aec89148d7b517cf (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240502 21:39:31.301523 4687 tablet_service.cc:1467] Processing CreateTablet for tablet ed2181a7e8194705934dadd2d4303b8a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 21:39:31.302886 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2181a7e8194705934dadd2d4303b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.303906 4838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 47) took 1102 ms. Trace:
I20240502 21:39:31.304615 4683 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 30) took 1001 ms. Trace:
I20240502 21:39:31.304325 4838 rpcz_store.cc:276] 0502 21:39:30.201293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.120006 (+918713us) service_pool.cc:225] Handling call
0502 21:39:31.165824 (+ 45818us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.165937 (+ 113us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.291718 (+125781us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.303312 (+ 11594us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.303840 (+ 528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.305051 4683 rpcz_store.cc:276] 0502 21:39:30.303493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:30.853548 (+550055us) service_pool.cc:225] Handling call
0502 21:39:30.939255 (+ 85707us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:30.939395 (+ 140us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.109272 (+169877us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.296548 (+187276us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.304552 (+ 8004us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.306970 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb1726ee5764d42806fc4c077226bf9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 21:39:31.309185 4895 tablet_bootstrap.cc:654] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.317736 4838 tablet_service.cc:1467] Processing CreateTablet for tablet df2394febdff4bb8bd495e4b8086e412 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 21:39:31.318403 4688 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65cfc531344398b70ce5862b2211fd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 21:39:31.320660 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240502 21:39:31.322458 4760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 21) took 1039 ms. Trace:
I20240502 21:39:31.324381 4689 tablet_service.cc:1467] Processing CreateTablet for tablet a06147be82454bb0a23d0f3e68c06f04 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 21:39:31.322855 4760 rpcz_store.cc:276] 0502 21:39:30.282946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:30.627942 (+344996us) service_pool.cc:225] Handling call
0502 21:39:30.823858 (+195916us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:30.823967 (+ 109us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.119007 (+295040us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.321989 (+202982us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.322374 (+ 385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.310081 4839 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 49) took 1108 ms. Trace:
I20240502 21:39:31.297852 4899 raft_consensus.cc:348] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.328488 4839 rpcz_store.cc:276] 0502 21:39:30.202079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.131170 (+929091us) service_pool.cc:225] Handling call
0502 21:39:31.166452 (+ 35282us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.166666 (+ 214us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.215105 (+ 48439us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.284019 (+ 68914us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.310014 (+ 25995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.329056 4899 raft_consensus.cc:374] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.329384 4899 raft_consensus.cc:729] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.291002 4842 tablet_service.cc:1467] Processing CreateTablet for tablet 9c974ff399fa4517af1e6d06b99af4f5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 21:39:31.316231 4908 raft_consensus.cc:482] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.309667 4896 tablet_bootstrap.cc:654] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.329536 4592 catalog_manager.cc:5484] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:31.330253 4908 raft_consensus.cc:504] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.333040 4908 leader_election.cc:288] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.337144 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.336933 4681 tablet_service.cc:1467] Processing CreateTablet for tablet f14af5e6ab984917b771589cdb45a779 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 21:39:31.338476 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14af5e6ab984917b771589cdb45a779. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:39:31.338595 4817 leader_election.cc:341] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.339008 4837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 46) took 1138 ms. Trace:
I20240502 21:39:31.336062 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.308197 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb1726ee5764d42806fc4c077226bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.339301 4840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d296f3d569f143fca949d49feb27d19d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:39:31.340415 4818 leader_election.cc:341] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.340085 4688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c65cfc531344398b70ce5862b2211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.340902 4818 leader_election.cc:302] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:31.342782 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a911c9636f426e9d08bee71f60eaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.342901 4908 raft_consensus.cc:2726] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.298203 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0fda9381a074590aec89148d7b517cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.349050 4839 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0e4a551d0f4f058d2d97346f8a247b (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 21:39:31.351891 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2fb6c68b641a0a4e1ab5356b23c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.355317 4759 tablet_service.cc:1467] Processing CreateTablet for tablet a1a911c9636f426e9d08bee71f60eaab (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240502 21:39:31.340912 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06147be82454bb0a23d0f3e68c06f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.357457 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c974ff399fa4517af1e6d06b99af4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.360635 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2394febdff4bb8bd495e4b8086e412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.362932 4764 tablet_service.cc:1467] Processing CreateTablet for tablet ed2181a7e8194705934dadd2d4303b8a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240502 21:39:31.353155 4689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06147be82454bb0a23d0f3e68c06f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.364468 4764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2181a7e8194705934dadd2d4303b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.322599 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 91e028b61d3e4a92b66697049b88e05a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 21:39:31.366991 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb1726ee5764d42806fc4c077226bf9 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240502 21:39:31.368258 4685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 34) took 1023 ms. Trace:
I20240502 21:39:31.370443 4680 tablet_service.cc:1467] Processing CreateTablet for tablet d296f3d569f143fca949d49feb27d19d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 21:39:31.372663 4839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0e4a551d0f4f058d2d97346f8a247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.339370 4837 rpcz_store.cc:276] 0502 21:39:30.200971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.071714 (+870743us) service_pool.cc:225] Handling call
0502 21:39:31.112674 (+ 40960us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.112782 (+ 108us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.127491 (+ 14709us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.338481 (+210990us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.338945 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84480}
I20240502 21:39:31.373430 4765 tablet_service.cc:1467] Processing CreateTablet for tablet 91e028b61d3e4a92b66697049b88e05a (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240502 21:39:31.374408 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14af5e6ab984917b771589cdb45a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.376035 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 1c65cfc531344398b70ce5862b2211fd (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240502 21:39:31.330145 4899 consensus_queue.cc:260] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.376746 4836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 44) took 1176 ms. Trace:
I20240502 21:39:31.379009 4685 rpcz_store.cc:276] 0502 21:39:30.345105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:30.935138 (+590033us) service_pool.cc:225] Handling call
0502 21:39:31.085411 (+150273us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.085542 (+ 131us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.225300 (+139758us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.347738 (+122438us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.368194 (+ 20456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.389917 4837 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c2c7e0cb241adb79d796752f2d4c6 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 21:39:31.391324 4837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c2c7e0cb241adb79d796752f2d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.396546 4895 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:31.396389 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 9c974ff399fa4517af1e6d06b99af4f5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 21:39:31.397488 4895 ts_tablet_manager.cc:1367] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.179s user 0.015s sys 0.000s
I20240502 21:39:31.397884 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c974ff399fa4517af1e6d06b99af4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.382908 4836 rpcz_store.cc:276] 0502 21:39:30.200232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.047733 (+847501us) service_pool.cc:225] Handling call
0502 21:39:31.085044 (+ 37311us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.113548 (+ 28504us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.221712 (+108164us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.334065 (+112353us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.376665 (+ 42600us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.402371 4763 tablet_service.cc:1467] Processing CreateTablet for tablet a06147be82454bb0a23d0f3e68c06f04 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240502 21:39:31.404414 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a911c9636f426e9d08bee71f60eaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.407243 4836 tablet_service.cc:1467] Processing CreateTablet for tablet bc0f282d043b43e8b9548ec320bdad33 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 21:39:31.410765 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb1726ee5764d42806fc4c077226bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.412212 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c65cfc531344398b70ce5862b2211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.419234 4899 ts_tablet_manager.cc:1398] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.125s user 0.005s sys 0.002s
I20240502 21:39:31.419019 4756 tablet_service.cc:1467] Processing CreateTablet for tablet d296f3d569f143fca949d49feb27d19d (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240502 21:39:31.420097 4896 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.420358 4899 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.420652 4896 ts_tablet_manager.cc:1367] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.155s user 0.016s sys 0.006s
I20240502 21:39:31.421962 4903 raft_consensus.cc:482] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.422483 4903 raft_consensus.cc:504] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.423777 4896 raft_consensus.cc:348] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.424564 4896 raft_consensus.cc:374] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.441632 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06147be82454bb0a23d0f3e68c06f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.443856 4895 raft_consensus.cc:348] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.444833 4895 raft_consensus.cc:374] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.445137 4895 raft_consensus.cc:729] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.445860 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d296f3d569f143fca949d49feb27d19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.446273 4835 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 48) took 1244 ms. Trace:
I20240502 21:39:31.446710 4835 rpcz_store.cc:276] 0502 21:39:30.201653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.122767 (+921114us) service_pool.cc:225] Handling call
0502 21:39:31.125821 (+ 3054us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.125932 (+ 111us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.306332 (+180400us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.445726 (+139394us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.446188 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.448544 4840 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 53) took 1128 ms. Trace:
I20240502 21:39:31.448832 4682 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 43) took 1050 ms. Trace:
I20240502 21:39:31.448957 4840 rpcz_store.cc:276] 0502 21:39:30.319760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.235851 (+916091us) service_pool.cc:225] Handling call
0502 21:39:31.275988 (+ 40137us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.276118 (+ 130us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.410507 (+134389us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.447978 (+ 37471us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.448477 (+ 499us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.449280 4682 rpcz_store.cc:276] 0502 21:39:30.397963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.147497 (+749534us) service_pool.cc:225] Handling call
0502 21:39:31.193481 (+ 45984us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.193586 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.343906 (+150320us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.448350 (+104444us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.448770 (+ 420us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.450577 4758 tablet_service.cc:1467] Processing CreateTablet for tablet f14af5e6ab984917b771589cdb45a779 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240502 21:39:31.451257 4682 tablet_service.cc:1467] Processing CreateTablet for tablet df2394febdff4bb8bd495e4b8086e412 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 21:39:31.451803 4765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e028b61d3e4a92b66697049b88e05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.427444 4903 leader_election.cc:288] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.453262 4766 tablet_service.cc:1467] Processing CreateTablet for tablet 9c974ff399fa4517af1e6d06b99af4f5 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240502 21:39:31.454705 4899 tablet_bootstrap.cc:654] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.462143 4836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0f282d043b43e8b9548ec320bdad33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.428015 4896 raft_consensus.cc:729] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.459434 4842 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 54) took 1139 ms. Trace:
I20240502 21:39:31.469758 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d296f3d569f143fca949d49feb27d19d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.480808 4896 consensus_queue.cc:260] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.484737 4903 raft_consensus.cc:482] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.485244 4903 raft_consensus.cc:504] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.487263 4903 leader_election.cc:288] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259), 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:31.488660 4681 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 52) took 1055 ms. Trace:
I20240502 21:39:31.489507 4684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 46) took 1090 ms. Trace:
I20240502 21:39:31.489953 4684 rpcz_store.cc:276] 0502 21:39:30.398924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.267490 (+868566us) service_pool.cc:225] Handling call
0502 21:39:31.276379 (+ 8889us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.298576 (+ 22197us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.403685 (+105109us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.483223 (+ 79538us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.489445 (+ 6222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.490937 4838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 55) took 1170 ms. Trace:
I20240502 21:39:31.491318 4838 rpcz_store.cc:276] 0502 21:39:30.320502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.316805 (+996303us) service_pool.cc:225] Handling call
0502 21:39:31.318659 (+ 1854us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.318778 (+ 119us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.441513 (+122735us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.490407 (+ 48894us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.490875 (+ 468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.489063 4681 rpcz_store.cc:276] 0502 21:39:30.433135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.335789 (+902654us) service_pool.cc:225] Handling call
0502 21:39:31.338331 (+ 2542us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.360483 (+ 22152us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.445675 (+ 85192us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.475216 (+ 29541us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.488590 (+ 13374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1700096}
I20240502 21:39:31.493666 4837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 57) took 1172 ms. Trace:
I20240502 21:39:31.493891 4762 tablet_service.cc:1467] Processing CreateTablet for tablet df2394febdff4bb8bd495e4b8086e412 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240502 21:39:31.494200 4837 rpcz_store.cc:276] 0502 21:39:30.321207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.388972 (+1067765us) service_pool.cc:225] Handling call
0502 21:39:31.390826 (+ 1854us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.390930 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.458572 (+ 67642us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.493031 (+ 34459us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.493580 (+ 549us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.494554 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c2c7e0cb241adb79d796752f2d4c6 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 21:39:31.495344 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2394febdff4bb8bd495e4b8086e412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.459403 4895 consensus_queue.cc:260] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.479876 4842 rpcz_store.cc:276] 0502 21:39:30.320112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.290028 (+969916us) service_pool.cc:225] Handling call
0502 21:39:31.341297 (+ 51269us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.341418 (+ 121us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.371284 (+ 29866us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.458881 (+ 87597us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.459352 (+ 471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.521827 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c2c7e0cb241adb79d796752f2d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.519994 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14af5e6ab984917b771589cdb45a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.529831 4833 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 52) took 1253 ms. Trace:
I20240502 21:39:31.530289 4833 rpcz_store.cc:276] 0502 21:39:30.319303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.233504 (+914201us) service_pool.cc:225] Handling call
0502 21:39:31.235327 (+ 1823us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.269902 (+ 34575us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.441878 (+171976us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.529291 (+ 87413us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.529763 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.535841 4839 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 56) took 1215 ms. Trace:
I20240502 21:39:31.536274 4839 rpcz_store.cc:276] 0502 21:39:30.320865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.329548 (+1008683us) service_pool.cc:225] Handling call
0502 21:39:31.349953 (+ 20405us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.350059 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.409414 (+ 59355us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.535295 (+125881us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.535776 (+ 481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.538674 4896 ts_tablet_manager.cc:1398] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.118s user 0.007s sys 0.001s
I20240502 21:39:31.539842 4896 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:31.541951 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c974ff399fa4517af1e6d06b99af4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.542368 4836 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 58) took 1220 ms. Trace:
I20240502 21:39:31.543524 4836 rpcz_store.cc:276] 0502 21:39:30.321517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.406295 (+1084778us) service_pool.cc:225] Handling call
0502 21:39:31.408131 (+ 1836us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.408221 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.528976 (+120755us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.540944 (+ 11968us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.542119 (+ 1175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.545387 4910 raft_consensus.cc:482] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.545926 4910 raft_consensus.cc:504] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.548004 4910 leader_election.cc:288] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.549275 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.549585 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.549996 4776 raft_consensus.cc:2454] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:31.550350 4841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 51) took 1347 ms. Trace:
W20240502 21:39:31.551142 4662 leader_election.cc:341] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.550704 4841 rpcz_store.cc:276] 0502 21:39:30.202731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.216448 (+1013717us) service_pool.cc:225] Handling call
0502 21:39:31.277794 (+ 61346us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.278033 (+ 239us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.484763 (+206730us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.549825 (+ 65062us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.550287 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1357056}
I20240502 21:39:31.551797 4664 leader_election.cc:302] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.552675 4687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 47) took 1153 ms. Trace:
I20240502 21:39:31.553171 4687 rpcz_store.cc:276] 0502 21:39:30.399307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.282243 (+882936us) service_pool.cc:225] Handling call
0502 21:39:31.302398 (+ 20155us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.302494 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.479320 (+176826us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.541412 (+ 62092us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.552579 (+ 11167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.555526 4910 raft_consensus.cc:2781] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.556066 4910 raft_consensus.cc:482] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.560500 4910 raft_consensus.cc:3037] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.560647 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2394febdff4bb8bd495e4b8086e412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.561239 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0e4a551d0f4f058d2d97346f8a247b (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 21:39:31.522447 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0e4a551d0f4f058d2d97346f8a247b (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240502 21:39:31.562577 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0e4a551d0f4f058d2d97346f8a247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.564638 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e028b61d3e4a92b66697049b88e05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.573859 4895 ts_tablet_manager.cc:1398] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.168s user 0.008s sys 0.001s
I20240502 21:39:31.574564 4834 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42764 (request call id 50) took 1372 ms. Trace:
I20240502 21:39:31.575081 4834 rpcz_store.cc:276] 0502 21:39:30.202411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.188460 (+986049us) service_pool.cc:225] Handling call
0502 21:39:31.190254 (+ 1794us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.190358 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.349369 (+159011us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.506947 (+157578us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.574478 (+ 67531us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13687}
I20240502 21:39:31.557268 4687 tablet_service.cc:1467] Processing CreateTablet for tablet bc0f282d043b43e8b9548ec320bdad33 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 21:39:31.562577 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a0e4a551d0f4f058d2d97346f8a247b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.575974 4895 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:31.600857 4687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0f282d043b43e8b9548ec320bdad33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.605273 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.605549 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:31.610224 4740 leader_election.cc:341] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.612001 4909 consensus_queue.cc:1035] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:31.618404 4684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 56) took 1134 ms. Trace:
I20240502 21:39:31.629581 4681 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 57) took 1145 ms. Trace:
I20240502 21:39:31.639554 4681 rpcz_store.cc:276] 0502 21:39:30.484144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.493631 (+1009487us) service_pool.cc:225] Handling call
0502 21:39:31.495719 (+ 2088us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.495825 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.571859 (+ 76034us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.622189 (+ 50330us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.628379 (+ 6190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.640463 4688 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 50) took 1239 ms. Trace:
I20240502 21:39:31.627905 4684 rpcz_store.cc:276] 0502 21:39:30.483836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.491913 (+1008077us) service_pool.cc:225] Handling call
0502 21:39:31.562116 (+ 70203us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.562213 (+ 97us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.572158 (+ 9945us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.617919 (+ 45761us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.618338 (+ 419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.656595 4685 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 54) took 1173 ms. Trace:
I20240502 21:39:31.656955 4685 rpcz_store.cc:276] 0502 21:39:30.483077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.387316 (+904239us) service_pool.cc:225] Handling call
0502 21:39:31.397373 (+ 10057us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.397486 (+ 113us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.649629 (+252143us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.656144 (+ 6515us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.656540 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.657439 4896 tablet_bootstrap.cc:654] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.643878 4688 rpcz_store.cc:276] 0502 21:39:30.432344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.317537 (+885193us) service_pool.cc:225] Handling call
0502 21:39:31.319199 (+ 1662us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.319426 (+ 227us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.383705 (+ 64279us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.633101 (+249396us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.639811 (+ 6710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.645433 4689 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 51) took 1211 ms. Trace:
I20240502 21:39:31.673660 4899 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.663367 4689 rpcz_store.cc:276] 0502 21:39:30.432814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.323374 (+890560us) service_pool.cc:225] Handling call
0502 21:39:31.326860 (+ 3486us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.327187 (+ 327us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.568381 (+241194us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.634644 (+ 66263us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.644178 (+ 9534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.675386 4687 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 58) took 1190 ms. Trace:
I20240502 21:39:31.675853 4687 rpcz_store.cc:276] 0502 21:39:30.493517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.555900 (+1062383us) service_pool.cc:225] Handling call
0502 21:39:31.594674 (+ 38774us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.600461 (+ 5787us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.668198 (+ 67737us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.674905 (+ 6707us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.675327 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.676831 4683 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 48) took 1277 ms. Trace:
I20240502 21:39:31.674198 4899 ts_tablet_manager.cc:1367] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.254s user 0.043s sys 0.023s
I20240502 21:39:31.677201 4683 rpcz_store.cc:276] 0502 21:39:30.399698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.306049 (+906351us) service_pool.cc:225] Handling call
0502 21:39:31.307747 (+ 1698us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.307844 (+ 97us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.467889 (+160045us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.676291 (+208402us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.676770 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.680776 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.681118 4899 raft_consensus.cc:348] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.682224 4899 raft_consensus.cc:374] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.680234 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.683101 4899 raft_consensus.cc:729] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
W20240502 21:39:31.683808 4737 leader_election.cc:341] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.684089 4899 consensus_queue.cc:260] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:31.684834 4737 leader_election.cc:341] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.685381 4737 leader_election.cc:302] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.687033 4899 ts_tablet_manager.cc:1398] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20240502 21:39:31.687875 4903 raft_consensus.cc:2726] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.688545 4899 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.694715 4899 tablet_bootstrap.cc:654] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.662271 4910 raft_consensus.cc:504] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.697588 4698 raft_consensus.cc:2379] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:31.698822 4910 leader_election.cc:288] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.699306 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:31.700488 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad5baefdf3c477b9ac99551bd3263ae" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
W20240502 21:39:31.700906 4662 leader_election.cc:341] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.701333 4776 raft_consensus.cc:3037] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.703508 4763 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 51) took 1113 ms. Trace:
I20240502 21:39:31.705145 4740 leader_election.cc:302] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.672641 4686 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 49) took 1272 ms. Trace:
I20240502 21:39:31.718021 4686 rpcz_store.cc:276] 0502 21:39:30.400049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.315036 (+914987us) service_pool.cc:225] Handling call
0502 21:39:31.516770 (+201734us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b5800005198. stack: 00007f1253fc2ba3 00007f1253fc2aa0 00007f1253927584 00007f125d63d9e4 00007f125d63d979 00007f1253e3bb51 00007f1253fbd3b1 00007f1253fbd9e2 00007f125ced85f9 00007f125bdec268 00007f125beabfb4 00007f125be5d619 00007f125545bc64 00007f125545bbf2 00007f125545bb21 00007f125545ba9c
0502 21:39:31.516837 (+ 67us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.516946 (+ 109us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.609879 (+ 92933us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.671909 (+ 62030us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.672545 (+ 636us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3617024}
I20240502 21:39:31.719821 4759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 46) took 1173 ms. Trace:
I20240502 21:39:31.720614 4895 tablet_bootstrap.cc:654] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.720520 4759 rpcz_store.cc:276] 0502 21:39:30.546058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.343807 (+797749us) service_pool.cc:225] Handling call
0502 21:39:31.356157 (+ 12350us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.356249 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.555569 (+199320us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.719288 (+163719us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.719757 (+ 469us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 21:39:31.717092 4928 log.cc:924] Time spent T 21a62e8a8c024197ab0afb8c0c284829 P df8e5a877b6640d692917083c9fb5282: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240502 21:39:31.719784 4896 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.722929 4896 ts_tablet_manager.cc:1367] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.183s user 0.009s sys 0.004s
I20240502 21:39:31.715459 4776 raft_consensus.cc:2454] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:31.726552 4903 raft_consensus.cc:2726] T fad5baefdf3c477b9ac99551bd3263ae P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.728507 4758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 52) took 1138 ms. Trace:
I20240502 21:39:31.728600 4664 leader_election.cc:302] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.729120 4758 rpcz_store.cc:276] 0502 21:39:30.590331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.417939 (+827608us) service_pool.cc:225] Handling call
0502 21:39:31.479599 (+ 61660us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.483544 (+ 3945us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.612078 (+128534us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.727766 (+115688us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.728419 (+ 653us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.730043 4909 raft_consensus.cc:2781] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:31.719326 4910 raft_consensus.cc:482] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.742905 4896 raft_consensus.cc:348] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.744570 4680 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 53) took 1270 ms. Trace:
I20240502 21:39:31.744392 4910 raft_consensus.cc:504] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.745152 4680 rpcz_store.cc:276] 0502 21:39:30.479158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.369394 (+890236us) service_pool.cc:225] Handling call
0502 21:39:31.384231 (+ 14837us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.405120 (+ 20889us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.623886 (+218766us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.743831 (+119945us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.744478 (+ 647us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.747262 4910 leader_election.cc:288] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.748488 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8321179b7548c08b34565f04746387" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:31.750006 4662 leader_election.cc:341] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.757800 4764 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 47) took 1211 ms. Trace:
I20240502 21:39:31.744033 4896 raft_consensus.cc:374] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.760542 4896 raft_consensus.cc:729] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.756850 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8321179b7548c08b34565f04746387" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.761608 4896 consensus_queue.cc:260] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.761952 4776 raft_consensus.cc:2454] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:31.758369 4764 rpcz_store.cc:276] 0502 21:39:30.588479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.361982 (+773503us) service_pool.cc:225] Handling call
0502 21:39:31.363929 (+ 1947us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.364034 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.688430 (+324396us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.740511 (+ 52081us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.757712 (+ 17201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.764091 4896 ts_tablet_manager.cc:1398] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20240502 21:39:31.763931 4664 leader_election.cc:302] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.765209 4766 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 54) took 1174 ms. Trace:
I20240502 21:39:31.704066 4763 rpcz_store.cc:276] 0502 21:39:30.589990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.401348 (+811358us) service_pool.cc:225] Handling call
0502 21:39:31.412073 (+ 10725us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.412194 (+ 121us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.596879 (+184685us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.702689 (+105810us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.703426 (+ 737us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.759935 4761 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 50) took 1170 ms. Trace:
I20240502 21:39:31.777230 4758 tablet_service.cc:1467] Processing CreateTablet for tablet bc0f282d043b43e8b9548ec320bdad33 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240502 21:39:31.777436 4910 consensus_queue.cc:1035] T 21a62e8a8c024197ab0afb8c0c284829 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:31.778641 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0f282d043b43e8b9548ec320bdad33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.780158 4765 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 49) took 1190 ms. Trace:
I20240502 21:39:31.780689 4765 rpcz_store.cc:276] 0502 21:39:30.589304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.372200 (+782896us) service_pool.cc:225] Handling call
0502 21:39:31.388949 (+ 16749us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800192998. stack: 00007f1253fc2ba3 00007f1253fc2aa0 00007f1253927584 00007f125d63d9e4 00007f125d63d979 00007f1253e3bb51 00007f1253fbd6dc 00007f1253fbd29b 00007f125ceaf889 00007f125bdec3c0 00007f125bece791 00007f125beabf03 00007f125be5d619 00007f125545bc64 00007f125545bbf2 00007f125545bb21
0502 21:39:31.391715 (+ 2766us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.391818 (+ 103us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.645617 (+253799us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.779637 (+134020us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.780092 (+ 455us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5956480}
I20240502 21:39:31.773841 4909 raft_consensus.cc:686] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:31.782888 4909 consensus_queue.cc:237] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:31.771598 4766 rpcz_store.cc:276] 0502 21:39:30.590978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.452290 (+861312us) service_pool.cc:225] Handling call
0502 21:39:31.510827 (+ 58537us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800192998. stack: 00007f1253fc2ba3 00007f1253fc2aa0 00007f1253927584 00007f125d63d9e4 00007f125d63d979 00007f1253e3bb51 00007f1253fbd3b1 00007f1253fbd262 00007f125ceaf889 00007f125bdec3c0 00007f125bece791 00007f125beabf03 00007f125be5d619 00007f125545bc64 00007f125545bbf2 00007f125545bb21
0502 21:39:31.510953 (+ 126us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.511055 (+ 102us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.552020 (+ 40965us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.764497 (+212477us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.765123 (+ 626us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6409600}
I20240502 21:39:31.786896 4932 raft_consensus.cc:2781] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.773368 4761 rpcz_store.cc:276] 0502 21:39:30.589646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.375106 (+785460us) service_pool.cc:225] Handling call
0502 21:39:31.389090 (+ 13984us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.389314 (+ 224us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.591500 (+202186us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.759205 (+167705us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.759845 (+ 640us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.787477 4932 raft_consensus.cc:482] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.787844 4932 raft_consensus.cc:3037] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.788936 4896 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:31.789858 4759 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3c2c7e0cb241adb79d796752f2d4c6 (DEFAULT_TABLE table=test-workload [id=b776127a918741d98edf5503b38d35bf]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240502 21:39:31.790674 4760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 56) took 1199 ms. Trace:
I20240502 21:39:31.791337 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3c2c7e0cb241adb79d796752f2d4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:31.794256 4762 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 55) took 1203 ms. Trace:
I20240502 21:39:31.794667 4756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 53) took 1204 ms. Trace:
I20240502 21:39:31.795336 4908 raft_consensus.cc:482] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.795528 4682 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53982 (request call id 55) took 1312 ms. Trace:
I20240502 21:39:31.791060 4760 rpcz_store.cc:276] 0502 21:39:30.591682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.560240 (+968558us) service_pool.cc:225] Handling call
0502 21:39:31.562116 (+ 1876us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.562213 (+ 97us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.751806 (+189593us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.790182 (+ 38376us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.790604 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.796288 4756 rpcz_store.cc:276] 0502 21:39:30.590665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.418027 (+827362us) service_pool.cc:225] Handling call
0502 21:39:31.419916 (+ 1889us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.420019 (+ 103us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.643382 (+223363us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.792972 (+149590us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.794552 (+ 1580us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.797456 4908 raft_consensus.cc:504] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.802089 4908 leader_election.cc:288] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.796192 4762 rpcz_store.cc:276] 0502 21:39:30.591328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.481532 (+890204us) service_pool.cc:225] Handling call
0502 21:39:31.494870 (+ 13338us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.494966 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.664436 (+169470us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.785362 (+120926us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.794169 (+ 8807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.803354 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.797441 4682 rpcz_store.cc:276] 0502 21:39:30.483471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.450377 (+966906us) service_pool.cc:225] Handling call
0502 21:39:31.479189 (+ 28812us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.482179 (+ 2990us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.653465 (+171286us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.794984 (+141519us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.795460 (+ 476us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240502 21:39:31.804975 4817 leader_election.cc:341] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.805742 4899 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.805585 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.806169 4899 ts_tablet_manager.cc:1367] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.118s user 0.022s sys 0.039s
W20240502 21:39:31.806855 4818 leader_election.cc:341] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.807330 4818 leader_election.cc:302] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:31.808127 4908 raft_consensus.cc:2726] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.808663 4899 raft_consensus.cc:348] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.811300 4899 raft_consensus.cc:374] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.811607 4899 raft_consensus.cc:729] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.812472 4899 consensus_queue.cc:260] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.815053 4899 ts_tablet_manager.cc:1398] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:39:31.816046 4899 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.817456 4932 raft_consensus.cc:504] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.819309 4758 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 58) took 1227 ms. Trace:
I20240502 21:39:31.819643 4903 raft_consensus.cc:482] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.822010 4899 tablet_bootstrap.cc:654] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.828632 4895 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:31.830226 4895 ts_tablet_manager.cc:1367] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.254s user 0.011s sys 0.004s
I20240502 21:39:31.830202 4903 raft_consensus.cc:504] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.831511 4896 tablet_bootstrap.cc:654] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.832340 4903 leader_election.cc:288] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.827852 4758 rpcz_store.cc:276] 0502 21:39:30.621982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.730398 (+1108416us) service_pool.cc:225] Handling call
0502 21:39:31.778094 (+ 47696us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.778211 (+ 117us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.811610 (+ 33399us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.818770 (+ 7160us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.819233 (+ 463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.836598 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8321179b7548c08b34565f04746387" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:31.836644 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.838127 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.841405 4775 raft_consensus.cc:3037] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.846518 4932 leader_election.cc:288] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.839936 4895 raft_consensus.cc:348] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.848583 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8321179b7548c08b34565f04746387" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
W20240502 21:39:31.848773 4737 leader_election.cc:341] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:31.852298 4740 leader_election.cc:341] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.852797 4740 leader_election.cc:302] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.853922 4903 raft_consensus.cc:2726] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.848125 4895 raft_consensus.cc:374] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.839965 4593 catalog_manager.cc:5484] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:31.860357 4895 raft_consensus.cc:729] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.861416 4895 consensus_queue.cc:260] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.865711 4895 ts_tablet_manager.cc:1398] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.035s user 0.004s sys 0.003s
W20240502 21:39:31.854945 4662 leader_election.cc:341] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.868080 4934 raft_consensus.cc:482] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.868678 4895 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:31.868525 4934 raft_consensus.cc:504] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.870674 4934 leader_election.cc:288] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.872445 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:31.874148 4662 leader_election.cc:341] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.877807 4895 tablet_bootstrap.cc:654] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.878330 4775 raft_consensus.cc:2454] T 8a8321179b7548c08b34565f04746387 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:31.879559 4664 leader_election.cc:302] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.879521 4759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:39510 (request call id 57) took 1287 ms. Trace:
I20240502 21:39:31.880458 4759 rpcz_store.cc:276] 0502 21:39:30.592041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:39:31.734176 (+1142135us) service_pool.cc:225] Handling call
0502 21:39:31.790772 (+ 56596us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0502 21:39:31.790878 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0502 21:39:31.850691 (+ 59813us) tablet_metadata.cc:641] Metadata flushed
0502 21:39:31.860137 (+ 9446us) tablet_replica.cc:205] Creating consensus instance
0502 21:39:31.879457 (+ 19320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240502 21:39:31.881080 4934 raft_consensus.cc:2781] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:31.881594 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.881970 4934 raft_consensus.cc:686] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:31.883566 4934 consensus_queue.cc:237] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:31.883996 4664 leader_election.cc:341] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.884660 4664 leader_election.cc:302] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:31.886739 4909 raft_consensus.cc:2726] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:31.891675 4896 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.893249 4896 ts_tablet_manager.cc:1367] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.007s
I20240502 21:39:31.897046 4896 raft_consensus.cc:348] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.898070 4896 raft_consensus.cc:374] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.898459 4896 raft_consensus.cc:729] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.899396 4896 consensus_queue.cc:260] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.906795 4899 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.907348 4899 ts_tablet_manager.cc:1367] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.091s user 0.015s sys 0.016s
I20240502 21:39:31.908171 4896 ts_tablet_manager.cc:1398] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.015s user 0.005s sys 0.003s
I20240502 21:39:31.910364 4899 raft_consensus.cc:348] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.911160 4899 raft_consensus.cc:374] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.911451 4899 raft_consensus.cc:729] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.912237 4899 consensus_queue.cc:260] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.914796 4899 ts_tablet_manager.cc:1398] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 21:39:31.915721 4896 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:31.915766 4899 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.917941 4908 raft_consensus.cc:482] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.918470 4908 raft_consensus.cc:504] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.919548 4895 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:31.920100 4895 ts_tablet_manager.cc:1367] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.005s
I20240502 21:39:31.920574 4908 leader_election.cc:288] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.922489 4899 tablet_bootstrap.cc:654] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.922489 4896 tablet_bootstrap.cc:654] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.923012 4895 raft_consensus.cc:348] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.924007 4895 raft_consensus.cc:374] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.924330 4895 raft_consensus.cc:729] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.924371 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:31.925083 4776 raft_consensus.cc:2454] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:31.925177 4895 consensus_queue.cc:260] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.926491 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.927470 4895 ts_tablet_manager.cc:1398] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
W20240502 21:39:31.928059 4818 leader_election.cc:341] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.928520 4895 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:31.929203 4817 leader_election.cc:302] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:31.932061 4947 raft_consensus.cc:2781] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.932457 4947 raft_consensus.cc:482] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.932792 4947 raft_consensus.cc:3037] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.934907 4895 tablet_bootstrap.cc:654] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.939157 4899 tablet_bootstrap.cc:492] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.939690 4899 ts_tablet_manager.cc:1367] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.006s
I20240502 21:39:31.941156 4592 catalog_manager.cc:5484] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:31.942462 4899 raft_consensus.cc:348] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.943300 4899 raft_consensus.cc:374] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.943603 4899 raft_consensus.cc:729] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.950562 4947 raft_consensus.cc:504] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.951654 4896 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.952204 4896 ts_tablet_manager.cc:1367] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.002s
I20240502 21:39:31.953526 4899 consensus_queue.cc:260] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.954176 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:31.955665 4903 raft_consensus.cc:482] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:31.955412 4896 raft_consensus.cc:348] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.956319 4896 raft_consensus.cc:374] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:39:31.957490 4818 leader_election.cc:341] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.956120 4903 raft_consensus.cc:504] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.958110 4896 raft_consensus.cc:729] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.960945 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:31.960623 4896 consensus_queue.cc:260] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.959415 4903 leader_election.cc:288] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.961202 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:31.962064 4852 raft_consensus.cc:2454] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
W20240502 21:39:31.962182 4740 leader_election.cc:341] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.962996 4896 ts_tablet_manager.cc:1398] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:39:31.963213 4737 leader_election.cc:302] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:31.963868 4896 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:31.964099 4903 raft_consensus.cc:2781] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:31.964538 4903 raft_consensus.cc:482] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:31.964897 4903 raft_consensus.cc:3037] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.964955 4899 ts_tablet_manager.cc:1398] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.025s user 0.006s sys 0.003s
I20240502 21:39:31.959911 4895 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:31.965950 4895 ts_tablet_manager.cc:1367] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.038s user 0.023s sys 0.007s
I20240502 21:39:31.967265 4899 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:31.970635 4896 tablet_bootstrap.cc:654] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.973966 4895 raft_consensus.cc:348] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.974903 4899 tablet_bootstrap.cc:654] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:31.975481 4903 raft_consensus.cc:504] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.977675 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047156e0297249998c3f2c5362c70432" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:31.978314 4776 raft_consensus.cc:3037] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.979916 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:31.980724 4852 raft_consensus.cc:3037] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:31.981510 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:31.982933 4740 leader_election.cc:341] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:31.983412 4895 raft_consensus.cc:374] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.984238 4895 raft_consensus.cc:729] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.985569 4895 consensus_queue.cc:260] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.988097 4895 ts_tablet_manager.cc:1398] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240502 21:39:31.988864 4899 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:31.988946 4776 raft_consensus.cc:2454] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:31.989504 4899 ts_tablet_manager.cc:1367] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.008s
I20240502 21:39:31.989774 4895 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:31.992259 4899 raft_consensus.cc:348] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.978523 4903 leader_election.cc:288] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:31.992715 4817 leader_election.cc:302] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:31.993120 4899 raft_consensus.cc:374] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:31.993536 4899 raft_consensus.cc:729] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:31.993865 4908 raft_consensus.cc:2781] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:31.994257 4899 consensus_queue.cc:260] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:31.996210 4852 raft_consensus.cc:2454] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:31.997642 4737 leader_election.cc:302] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:31.998538 4896 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:31.976691 4947 leader_election.cc:288] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:31.998908 4903 raft_consensus.cc:2781] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:31.999081 4896 ts_tablet_manager.cc:1367] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.000s
I20240502 21:39:31.999451 4903 raft_consensus.cc:686] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:32.000888 4903 consensus_queue.cc:237] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.002812 4896 raft_consensus.cc:348] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.003571 4896 raft_consensus.cc:374] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.003926 4896 raft_consensus.cc:729] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.004700 4896 consensus_queue.cc:260] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.009367 4899 ts_tablet_manager.cc:1398] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20240502 21:39:32.009771 4908 raft_consensus.cc:686] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:32.010772 4908 consensus_queue.cc:237] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.011410 4899 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.010792 4896 ts_tablet_manager.cc:1398] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 21:39:32.015903 4895 tablet_bootstrap.cc:654] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.017771 4896 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.022624 4899 tablet_bootstrap.cc:654] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.028327 4896 tablet_bootstrap.cc:654] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.032223 4899 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.032791 4899 ts_tablet_manager.cc:1367] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.000s
I20240502 21:39:32.037914 4895 tablet_bootstrap.cc:492] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.038393 4895 ts_tablet_manager.cc:1367] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.003s
I20240502 21:39:32.035475 4899 raft_consensus.cc:348] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.039974 4908 raft_consensus.cc:482] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.040374 4899 raft_consensus.cc:374] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.040510 4908 raft_consensus.cc:504] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.040848 4899 raft_consensus.cc:729] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.038394 4594 catalog_manager.cc:5484] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:32.041646 4899 consensus_queue.cc:260] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.041946 4895 raft_consensus.cc:348] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:32.042572 4895 raft_consensus.cc:374] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.042855 4895 raft_consensus.cc:729] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.043473 4895 consensus_queue.cc:260] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } }
I20240502 21:39:32.044159 4899 ts_tablet_manager.cc:1398] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 21:39:32.045080 4899 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.045863 4895 ts_tablet_manager.cc:1398] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:39:32.046751 4895 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.031647 4591 catalog_manager.cc:5484] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.050689 4899 tablet_bootstrap.cc:654] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.055642 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:32.057152 4818 leader_election.cc:341] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.057989 4895 tablet_bootstrap.cc:654] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.058081 4952 raft_consensus.cc:482] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.058621 4952 raft_consensus.cc:504] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.060842 4896 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.061137 4952 leader_election.cc:288] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.061362 4896 ts_tablet_manager.cc:1367] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.044s user 0.022s sys 0.014s
I20240502 21:39:32.062753 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.063416 4698 raft_consensus.cc:2454] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:32.064087 4896 raft_consensus.cc:348] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.064632 4740 leader_election.cc:302] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:32.064972 4896 raft_consensus.cc:374] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.065265 4896 raft_consensus.cc:729] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.065618 4952 raft_consensus.cc:2781] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.065973 4952 raft_consensus.cc:482] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.066366 4952 raft_consensus.cc:3037] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.066051 4896 consensus_queue.cc:260] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.066653 4908 leader_election.cc:288] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.069145 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.069468 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.070288 4852 raft_consensus.cc:2454] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:32.071377 4896 ts_tablet_manager.cc:1398] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
W20240502 21:39:32.071986 4817 leader_election.cc:341] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.072372 4896 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.072500 4817 leader_election.cc:302] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.077884 4895 tablet_bootstrap.cc:492] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.078095 4896 tablet_bootstrap.cc:654] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.078547 4895 ts_tablet_manager.cc:1367] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.007s
I20240502 21:39:32.080639 4934 raft_consensus.cc:482] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.081131 4934 raft_consensus.cc:504] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.083724 4899 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.084233 4899 ts_tablet_manager.cc:1367] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.004s
I20240502 21:39:32.086299 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.087232 4776 raft_consensus.cc:2454] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.086865 4899 raft_consensus.cc:348] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.088194 4899 raft_consensus.cc:374] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.087281 4895 raft_consensus.cc:348] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.088591 4899 raft_consensus.cc:729] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.088903 4895 raft_consensus.cc:374] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.088733 4664 leader_election.cc:302] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:32.087641 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.089222 4895 raft_consensus.cc:729] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.089830 4909 raft_consensus.cc:2781] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.092957 4909 raft_consensus.cc:482] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.093299 4909 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.093216 4895 consensus_queue.cc:260] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.095455 4895 ts_tablet_manager.cc:1398] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 21:39:32.096609 4895 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.101900 4895 tablet_bootstrap.cc:654] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.101567 4909 raft_consensus.cc:504] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:32.090620 4662 leader_election.cc:341] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.092465 4952 raft_consensus.cc:504] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.104512 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.086894 4908 raft_consensus.cc:2726] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.105268 4776 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.089251 4934 leader_election.cc:288] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.107087 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:32.107906 4698 raft_consensus.cc:3037] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.111124 4776 raft_consensus.cc:2454] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.089576 4899 consensus_queue.cc:260] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.112198 4664 leader_election.cc:302] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:32.090080 4896 tablet_bootstrap.cc:492] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.113077 4896 ts_tablet_manager.cc:1367] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.003s
I20240502 21:39:32.113145 4934 raft_consensus.cc:2781] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.113708 4934 raft_consensus.cc:686] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.114584 4934 consensus_queue.cc:237] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.115890 4895 tablet_bootstrap.cc:492] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.117445 4899 ts_tablet_manager.cc:1398] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20240502 21:39:32.118902 4895 ts_tablet_manager.cc:1367] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.008s
I20240502 21:39:32.119468 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.120038 4909 leader_election.cc:288] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.117358 4698 raft_consensus.cc:2454] T f4d1e2fc15594054a6cd01aab641f8cf P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:32.116036 4896 raft_consensus.cc:348] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.122397 4896 raft_consensus.cc:374] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:39:32.122452 4662 leader_election.cc:341] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.122967 4899 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.122774 4740 leader_election.cc:302] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:32.123333 4952 leader_election.cc:288] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.122810 4896 raft_consensus.cc:729] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.123940 4952 raft_consensus.cc:2781] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.124195 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d1e2fc15594054a6cd01aab641f8cf" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.124490 4952 raft_consensus.cc:686] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:32.124881 4852 raft_consensus.cc:3037] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.125350 4952 consensus_queue.cc:237] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.124071 4895 raft_consensus.cc:348] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.126622 4895 raft_consensus.cc:374] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.124509 4896 consensus_queue.cc:260] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.130924 4895 raft_consensus.cc:729] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.129787 4909 raft_consensus.cc:482] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.132359 4896 ts_tablet_manager.cc:1398] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240502 21:39:32.131770 4909 raft_consensus.cc:504] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.131755 4895 consensus_queue.cc:260] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.133282 4896 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.134464 4895 ts_tablet_manager.cc:1398] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 21:39:32.135370 4899 tablet_bootstrap.cc:654] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.138695 4896 tablet_bootstrap.cc:654] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.136329 4895 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.138860 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.139547 4850 raft_consensus.cc:2454] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.140988 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.140969 4662 leader_election.cc:302] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:32.142015 4909 leader_election.cc:288] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:32.143158 4664 leader_election.cc:341] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.145702 4909 raft_consensus.cc:2781] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.146366 4909 raft_consensus.cc:482] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.147184 4909 raft_consensus.cc:3037] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.148033 4592 catalog_manager.cc:5484] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.151269 4852 raft_consensus.cc:2454] T f4d1e2fc15594054a6cd01aab641f8cf P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:32.155179 4909 raft_consensus.cc:504] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.154933 4591 catalog_manager.cc:5484] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.158370 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
W20240502 21:39:32.159766 4664 leader_election.cc:341] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.159935 4895 tablet_bootstrap.cc:654] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.161154 4909 leader_election.cc:288] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.164252 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.164942 4852 raft_consensus.cc:3037] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.172869 4852 raft_consensus.cc:2454] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.174610 4662 leader_election.cc:302] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.177973 4909 raft_consensus.cc:2781] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.178524 4909 raft_consensus.cc:686] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.179495 4909 consensus_queue.cc:237] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.184217 4895 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.184741 4895 ts_tablet_manager.cc:1367] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.013s
I20240502 21:39:32.187443 4895 raft_consensus.cc:348] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.188172 4895 raft_consensus.cc:374] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.188482 4895 raft_consensus.cc:729] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.189167 4895 consensus_queue.cc:260] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.191193 4895 ts_tablet_manager.cc:1398] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:39:32.192461 4895 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.193220 4591 catalog_manager.cc:5484] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.197903 4895 tablet_bootstrap.cc:654] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.198661 4899 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.199188 4899 ts_tablet_manager.cc:1367] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.077s user 0.022s sys 0.007s
I20240502 21:39:32.202324 4908 raft_consensus.cc:482] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.203248 4908 raft_consensus.cc:504] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.203264 4899 raft_consensus.cc:348] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.204262 4899 raft_consensus.cc:374] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.204586 4899 raft_consensus.cc:729] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.205320 4909 raft_consensus.cc:482] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.205826 4909 raft_consensus.cc:504] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.209123 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.209702 4852 raft_consensus.cc:2454] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.210280 4909 leader_election.cc:288] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.210127 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.210755 4662 leader_election.cc:302] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:32.211122 4698 raft_consensus.cc:2454] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:32.211010 4899 consensus_queue.cc:260] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.211870 4909 raft_consensus.cc:2781] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.212284 4909 raft_consensus.cc:482] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.212693 4909 raft_consensus.cc:3037] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.213291 4899 ts_tablet_manager.cc:1398] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 21:39:32.213331 4818 leader_election.cc:302] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:32.214171 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.214501 4899 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.214470 4951 raft_consensus.cc:2781] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.215587 4951 raft_consensus.cc:482] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 21:39:32.215627 4664 leader_election.cc:341] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.216053 4951 raft_consensus.cc:3037] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.221098 4899 tablet_bootstrap.cc:654] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.222286 4908 leader_election.cc:288] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.222465 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.223224 4776 raft_consensus.cc:2454] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:32.224426 4909 raft_consensus.cc:504] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.227643 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
W20240502 21:39:32.229359 4664 leader_election.cc:341] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.230116 4896 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.230607 4896 ts_tablet_manager.cc:1367] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.016s
I20240502 21:39:32.231565 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.232154 4852 raft_consensus.cc:3037] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.233227 4896 raft_consensus.cc:348] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.233924 4896 raft_consensus.cc:374] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.234185 4896 raft_consensus.cc:729] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.235124 4896 consensus_queue.cc:260] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.236726 4951 raft_consensus.cc:504] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.240677 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:32.241134 4896 ts_tablet_manager.cc:1398] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20240502 21:39:32.241557 4698 raft_consensus.cc:3037] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.242141 4896 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.255553 4896 tablet_bootstrap.cc:654] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.255950 4909 leader_election.cc:288] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.258620 4852 raft_consensus.cc:2454] T 071379d3b0d24e4e9cfcb3f4cb2670fe P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.261144 4662 leader_election.cc:302] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.262905 4909 raft_consensus.cc:2781] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.263893 4909 raft_consensus.cc:686] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.265090 4909 consensus_queue.cc:237] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.267678 4896 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.268141 4896 ts_tablet_manager.cc:1367] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.011s
I20240502 21:39:32.265275 4698 raft_consensus.cc:2454] T b26fb0cda47f46d0b3fa153864652df0 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:32.268980 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26fb0cda47f46d0b3fa153864652df0" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.270817 4776 raft_consensus.cc:3037] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.271072 4818 leader_election.cc:302] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:32.272419 4908 raft_consensus.cc:2781] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.273064 4908 raft_consensus.cc:686] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:32.274199 4908 consensus_queue.cc:237] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.269076 4951 leader_election.cc:288] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.283536 4895 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.284101 4895 ts_tablet_manager.cc:1367] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.092s user 0.033s sys 0.048s
I20240502 21:39:32.292644 4934 consensus_queue.cc:1035] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.293789 4896 raft_consensus.cc:348] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.294767 4896 raft_consensus.cc:374] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.295099 4896 raft_consensus.cc:729] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.297461 4934 raft_consensus.cc:482] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.297962 4934 raft_consensus.cc:504] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.298002 4896 consensus_queue.cc:260] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.303042 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.302950 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf3772d51c64deaa4bc7c0d8e8e6c9a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.303926 4852 raft_consensus.cc:2379] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:32.306684 4896 ts_tablet_manager.cc:1398] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.038s user 0.009s sys 0.000s
I20240502 21:39:32.307991 4896 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.308488 4664 leader_election.cc:302] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.309365 4932 raft_consensus.cc:3037] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.312968 4776 raft_consensus.cc:2454] T b26fb0cda47f46d0b3fa153864652df0 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:32.300913 4934 leader_election.cc:288] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:32.316072 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:32.318301 4895 raft_consensus.cc:348] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.319064 4895 raft_consensus.cc:374] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.319341 4895 raft_consensus.cc:729] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.318324 4932 raft_consensus.cc:2726] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.320045 4895 consensus_queue.cc:260] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.322221 4895 ts_tablet_manager.cc:1398] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20240502 21:39:32.323144 4895 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.325170 4934 consensus_queue.cc:1035] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:32.328039 4899 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.328532 4899 ts_tablet_manager.cc:1367] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.114s user 0.027s sys 0.020s
I20240502 21:39:32.331632 4896 tablet_bootstrap.cc:654] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.332653 4591 catalog_manager.cc:5484] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:32.333822 4899 raft_consensus.cc:348] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.334519 4899 raft_consensus.cc:374] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.334791 4899 raft_consensus.cc:729] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.335475 4899 consensus_queue.cc:260] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.337633 4899 ts_tablet_manager.cc:1398] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:39:32.338506 4899 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
W20240502 21:39:32.343614 4662 consensus_peers.cc:485] T 8a8321179b7548c08b34565f04746387 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Couldn't send request to peer df8e5a877b6640d692917083c9fb5282. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:32.349376 4895 tablet_bootstrap.cc:654] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.354066 4899 tablet_bootstrap.cc:654] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.369880 4852 raft_consensus.cc:3037] T fad5baefdf3c477b9ac99551bd3263ae P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.398118 4895 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.398636 4895 ts_tablet_manager.cc:1367] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.008s
I20240502 21:39:32.397922 4592 catalog_manager.cc:5484] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.401408 4895 raft_consensus.cc:348] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.402151 4895 raft_consensus.cc:374] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.402427 4895 raft_consensus.cc:729] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.402961 4895 consensus_queue.cc:260] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.405130 4895 ts_tablet_manager.cc:1398] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:39:32.406682 4895 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.407918 4899 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.408401 4899 ts_tablet_manager.cc:1367] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.004s
I20240502 21:39:32.404533 4979 consensus_queue.cc:1035] T fad5baefdf3c477b9ac99551bd3263ae P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 21:39:32.411603 4896 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.412082 4896 ts_tablet_manager.cc:1367] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.006s
I20240502 21:39:32.413362 4895 tablet_bootstrap.cc:654] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.415845 4899 raft_consensus.cc:348] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.415812 4896 raft_consensus.cc:348] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.416888 4899 raft_consensus.cc:374] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.417014 4896 raft_consensus.cc:374] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.417222 4899 raft_consensus.cc:729] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.417430 4896 raft_consensus.cc:729] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.418614 4896 consensus_queue.cc:260] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.419234 4899 consensus_queue.cc:260] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.427556 4896 ts_tablet_manager.cc:1398] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240502 21:39:32.428560 4896 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.430641 4952 raft_consensus.cc:482] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.431335 4952 raft_consensus.cc:504] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.434906 4896 tablet_bootstrap.cc:654] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.435108 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.437088 4952 leader_election.cc:288] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.438117 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856dc0c130c04f4894884164ae24a053" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.443274 4698 raft_consensus.cc:3037] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.443775 4850 raft_consensus.cc:2379] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:32.441362 4895 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.444926 4737 leader_election.cc:302] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.445555 4895 ts_tablet_manager.cc:1367] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.015s
I20240502 21:39:32.441277 4979 raft_consensus.cc:482] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.446698 4979 raft_consensus.cc:504] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.447866 4952 raft_consensus.cc:3037] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.448491 4932 raft_consensus.cc:482] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.448977 4932 raft_consensus.cc:504] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.450119 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.450409 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.450815 4776 raft_consensus.cc:2454] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.450989 4932 leader_election.cc:288] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.452440 4664 leader_election.cc:302] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:32.452919 4908 raft_consensus.cc:482] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:39:32.453629 4662 leader_election.cc:341] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.453399 4908 raft_consensus.cc:504] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.445915 4899 ts_tablet_manager.cc:1398] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20240502 21:39:32.454932 4899 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.454885 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.455446 4908 leader_election.cc:288] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:32.458140 4664 leader_election.cc:341] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.458843 4932 raft_consensus.cc:2781] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.459239 4932 raft_consensus.cc:482] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.459607 4932 raft_consensus.cc:3037] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.448956 4895 raft_consensus.cc:348] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.460628 4895 raft_consensus.cc:374] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.460935 4895 raft_consensus.cc:729] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.463443 4899 tablet_bootstrap.cc:654] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.464900 4895 consensus_queue.cc:260] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.464535 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.466054 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.466354 4850 raft_consensus.cc:2454] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.461110 4979 leader_election.cc:288] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.467526 4895 ts_tablet_manager.cc:1398] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
W20240502 21:39:32.467406 4818 leader_election.cc:341] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.468412 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:32.470275 4817 leader_election.cc:341] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.470710 4817 leader_election.cc:302] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.471426 4908 raft_consensus.cc:2726] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.472364 4895 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.472744 4662 leader_election.cc:302] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.473918 4979 raft_consensus.cc:2781] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.474303 4979 raft_consensus.cc:482] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.474642 4979 raft_consensus.cc:3037] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.485993 4979 raft_consensus.cc:504] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.487857 4908 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:32.488231 4952 raft_consensus.cc:2726] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.475522 4932 raft_consensus.cc:504] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.491633 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.492072 4932 leader_election.cc:288] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:32.493643 4662 leader_election.cc:341] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.493492 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.494376 4776 raft_consensus.cc:3037] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.495257 4899 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.495723 4899 ts_tablet_manager.cc:1367] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.007s
I20240502 21:39:32.497290 4952 consensus_queue.cc:1035] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.524160 4979 leader_election.cc:288] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.549621 4952 consensus_queue.cc:1035] T 3bf3772d51c64deaa4bc7c0d8e8e6c9a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240502 21:39:32.507086 4908 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20240502 21:39:32.507539 4776 raft_consensus.cc:2454] T ba7a6cb1d5034911923934eee0d68132 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.528786 4852 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.563097 4664 leader_election.cc:302] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.564498 4979 raft_consensus.cc:2781] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.496879 4895 tablet_bootstrap.cc:654] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.565343 4979 raft_consensus.cc:686] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.566283 4979 consensus_queue.cc:237] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.513442 4951 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.568575 4896 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.568503 4951 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.569702 4896 ts_tablet_manager.cc:1367] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.141s user 0.031s sys 0.020s
I20240502 21:39:32.510195 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.568168 4952 consensus_queue.cc:1035] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:32.498504 4903 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.571518 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.571758 4903 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:32.574286 4664 leader_election.cc:341] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.576388 4990 raft_consensus.cc:482] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.577030 4990 raft_consensus.cc:504] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:32.578016 4818 leader_election.cc:341] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.578203 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.578025 4951 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.498263 4899 raft_consensus.cc:348] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.579380 4899 raft_consensus.cc:374] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:39:32.579618 4740 leader_election.cc:341] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.574443 4903 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.579684 4899 raft_consensus.cc:729] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.581467 4899 consensus_queue.cc:260] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.583691 4899 ts_tablet_manager.cc:1398] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.088s user 0.005s sys 0.001s
I20240502 21:39:32.560395 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.584326 4850 raft_consensus.cc:3037] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.587651 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.589758 4990 leader_election.cc:288] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.591820 4947 raft_consensus.cc:482] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.592485 4947 raft_consensus.cc:504] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:32.594246 4818 leader_election.cc:341] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.596295 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.597035 4849 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:32.597025 4947 leader_election.cc:288] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.598908 4896 raft_consensus.cc:348] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.599640 4896 raft_consensus.cc:374] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.599906 4896 raft_consensus.cc:729] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.600363 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.601537 4934 consensus_queue.cc:1035] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.603093 4896 consensus_queue.cc:260] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.611387 4903 raft_consensus.cc:482] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.609440 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.612568 4899 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.612433 4774 raft_consensus.cc:2454] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:32.612051 4903 raft_consensus.cc:504] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.616283 4896 ts_tablet_manager.cc:1398] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.046s user 0.003s sys 0.004s
I20240502 21:39:32.617520 4896 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.619127 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.619976 4773 raft_consensus.cc:2454] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:32.620510 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.631297 4776 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:32.619112 4993 raft_consensus.cc:482] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.636001 4934 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:32.637488 4896 tablet_bootstrap.cc:654] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.630728 4737 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.639078 4993 raft_consensus.cc:504] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.643077 4895 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.643563 4895 ts_tablet_manager.cc:1367] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.171s user 0.037s sys 0.017s
I20240502 21:39:32.645992 4895 raft_consensus.cc:348] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.646682 4895 raft_consensus.cc:374] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.646896 4934 raft_consensus.cc:482] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.646963 4895 raft_consensus.cc:729] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.647522 4934 raft_consensus.cc:504] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.648011 4895 consensus_queue.cc:260] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.637526 4903 leader_election.cc:288] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.649379 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.650483 4895 ts_tablet_manager.cc:1398] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:39:32.652887 4895 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
W20240502 21:39:32.654373 4740 leader_election.cc:341] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.654280 4817 leader_election.cc:302] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.655984 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.655613 4591 catalog_manager.cc:5484] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.657217 4990 raft_consensus.cc:2781] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.658303 4990 raft_consensus.cc:482] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.659116 4990 raft_consensus.cc:3037] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.660537 4934 leader_election.cc:288] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.643546 4993 leader_election.cc:288] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.666406 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.666853 4997 raft_consensus.cc:2781] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.667272 4997 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.667412 4849 raft_consensus.cc:2454] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:32.667675 4997 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.668507 4662 leader_election.cc:302] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:32.669327 4932 raft_consensus.cc:2781] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.669695 4932 raft_consensus.cc:482] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.670073 4932 raft_consensus.cc:3037] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.669756 4817 leader_election.cc:302] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.671298 4947 raft_consensus.cc:2781] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:39:32.671643 4947 raft_consensus.cc:482] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09ff1ee782864ac99d05a7f011aeef44)
I20240502 21:39:32.672021 4947 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:32.674706 4817 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.675135 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.675539 4908 raft_consensus.cc:2781] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:32.675913 4908 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:32.676249 4908 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:32.673805 4740 leader_election.cc:341] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.679482 4951 raft_consensus.cc:482] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.680122 4951 raft_consensus.cc:504] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.682205 4951 leader_election.cc:288] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.683538 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.682613 4908 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.684494 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.686043 4908 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:32.686659 4818 leader_election.cc:341] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.682473 4990 raft_consensus.cc:504] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.687927 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:32.688858 4990 leader_election.cc:288] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:32.689934 4818 leader_election.cc:341] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:32.690276 4664 leader_election.cc:341] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.690745 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:32.679504 4947 raft_consensus.cc:504] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:32.692296 4818 leader_election.cc:341] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.692045 4849 raft_consensus.cc:1226] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Rejecting Update request from peer 09ff1ee782864ac99d05a7f011aeef44 for earlier term 1. Current term is 2. Ops: []
I20240502 21:39:32.693018 4947 leader_election.cc:288] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:32.695252 4664 consensus_peers.cc:485] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 -> Peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Couldn't send request to peer ddea410051614f23b93c7342ec7cc545. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:32.696022 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:32.699074 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.699748 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.700397 4934 consensus_queue.cc:1035] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.701123 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.705413 4773 raft_consensus.cc:3037] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.707832 4895 tablet_bootstrap.cc:654] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.702766 4899 tablet_bootstrap.cc:654] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.702353 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:32.702497 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5e2e41273949e2afa8bc07812c9ca7" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
W20240502 21:39:32.703311 4737 leader_election.cc:341] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.681644 4850 raft_consensus.cc:2454] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.680116 4932 raft_consensus.cc:504] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.710947 4774 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:32.711273 4737 leader_election.cc:302] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.714149 4993 raft_consensus.cc:2726] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:39:32.714485 4737 leader_election.cc:341] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.714943 4737 leader_election.cc:302] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.716392 4993 raft_consensus.cc:2726] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.715986 4932 leader_election.cc:288] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.721165 4903 consensus_queue.cc:1035] T f4d1e2fc15594054a6cd01aab641f8cf P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.723034 4662 leader_election.cc:302] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.724385 4909 raft_consensus.cc:2781] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.724871 4909 raft_consensus.cc:686] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.725674 4909 consensus_queue.cc:237] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.727105 4979 consensus_queue.cc:1046] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:39:32.729151 4817 leader_election.cc:341] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.729630 4817 leader_election.cc:302] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.733693 4990 raft_consensus.cc:2726] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.734465 4773 raft_consensus.cc:2454] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:32.740967 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:32.741163 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.741832 4852 raft_consensus.cc:3037] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:32.747058 4817 leader_election.cc:302] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.750519 4947 raft_consensus.cc:2781] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.751667 4947 raft_consensus.cc:686] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
W20240502 21:39:32.752570 4664 leader_election.cc:341] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.754866 4975 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:32.759088 4774 raft_consensus.cc:2454] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 2.
I20240502 21:39:32.753105 4947 consensus_queue.cc:237] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.765085 4977 raft_consensus.cc:3032] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:32.765790 4977 raft_consensus.cc:729] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.765843 4817 leader_election.cc:302] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:32.767347 4990 raft_consensus.cc:2781] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:32.767138 4977 consensus_queue.cc:260] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.768055 4990 raft_consensus.cc:686] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [term 2 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:32.758870 4997 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.782703 4776 raft_consensus.cc:2379] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:32.786881 4990 consensus_queue.cc:237] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.788614 4896 tablet_bootstrap.cc:492] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.789410 4896 ts_tablet_manager.cc:1367] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.172s user 0.023s sys 0.012s
I20240502 21:39:32.794492 4997 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.796368 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:32.797755 4850 raft_consensus.cc:2379] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:32.803913 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:32.805917 4740 leader_election.cc:341] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.806531 4740 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.807317 4852 raft_consensus.cc:2454] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:32.808748 4903 raft_consensus.cc:2726] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.809129 4662 leader_election.cc:302] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.809530 4896 raft_consensus.cc:348] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.810225 4896 raft_consensus.cc:374] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.810657 4896 raft_consensus.cc:729] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.815704 4896 consensus_queue.cc:260] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.817448 5001 raft_consensus.cc:2781] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:32.846094 4776 raft_consensus.cc:1226] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Rejecting Update request from peer 09ff1ee782864ac99d05a7f011aeef44 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240502 21:39:32.849164 4977 raft_consensus.cc:3037] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:32.856406 5001 raft_consensus.cc:686] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:32.857427 5001 consensus_queue.cc:237] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.863130 4896 ts_tablet_manager.cc:1398] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
I20240502 21:39:32.861528 4817 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.868254 4908 raft_consensus.cc:2726] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.861428 4990 consensus_queue.cc:1035] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 21:39:32.876506 4899 tablet_bootstrap.cc:492] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.877142 4899 ts_tablet_manager.cc:1367] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.265s user 0.010s sys 0.004s
I20240502 21:39:32.874925 4896 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.880368 4593 catalog_manager.cc:5484] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:32.880431 4899 raft_consensus.cc:348] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.885128 4899 raft_consensus.cc:374] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.885404 4899 raft_consensus.cc:729] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.885855 4951 consensus_queue.cc:1035] T b26fb0cda47f46d0b3fa153864652df0 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.886224 4899 consensus_queue.cc:260] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.891669 4895 tablet_bootstrap.cc:492] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:32.892262 4895 ts_tablet_manager.cc:1367] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.240s user 0.007s sys 0.012s
I20240502 21:39:32.893146 4909 consensus_queue.cc:1035] T 071379d3b0d24e4e9cfcb3f4cb2670fe P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:32.897059 4592 catalog_manager.cc:5484] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.899775 4997 raft_consensus.cc:482] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.900429 4997 raft_consensus.cc:504] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.903966 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.907163 4997 leader_election.cc:288] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:32.909222 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "071379d3b0d24e4e9cfcb3f4cb2670fe" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:32.909605 4895 raft_consensus.cc:348] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.914232 4895 raft_consensus.cc:374] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.914593 4895 raft_consensus.cc:729] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.914721 4737 leader_election.cc:302] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:32.913275 4896 tablet_bootstrap.cc:654] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.916154 4997 raft_consensus.cc:2726] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.911067 4899 ts_tablet_manager.cc:1398] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20240502 21:39:32.915508 4895 consensus_queue.cc:260] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.922533 4895 ts_tablet_manager.cc:1398] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20240502 21:39:32.923951 4895 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:32.932643 4951 raft_consensus.cc:482] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.932898 4899 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.933185 4951 raft_consensus.cc:504] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.935530 4951 leader_election.cc:288] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:32.938040 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:32.939039 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:32.939569 4818 leader_election.cc:341] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.939565 4899 tablet_bootstrap.cc:654] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.942891 4896 tablet_bootstrap.cc:492] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.943388 4896 ts_tablet_manager.cc:1367] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.000s
W20240502 21:39:32.944768 4817 leader_election.cc:341] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:32.945271 4817 leader_election.cc:302] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:32.946136 4896 raft_consensus.cc:348] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.946503 4951 raft_consensus.cc:2726] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:32.946898 4896 raft_consensus.cc:374] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.947221 4896 raft_consensus.cc:729] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.943328 4592 catalog_manager.cc:5484] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 1 to 2, leader changed from 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 2 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:32.949012 4896 consensus_queue.cc:260] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.950644 4896 ts_tablet_manager.cc:1398] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:39:32.951326 4896 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.955204 4895 tablet_bootstrap.cc:654] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.955626 4896 tablet_bootstrap.cc:654] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.957132 4899 tablet_bootstrap.cc:492] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.957682 4899 ts_tablet_manager.cc:1367] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.003s
I20240502 21:39:32.960942 4899 raft_consensus.cc:348] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.962137 4899 raft_consensus.cc:374] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.962587 4899 raft_consensus.cc:729] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.963785 4899 consensus_queue.cc:260] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.964618 4591 catalog_manager.cc:5484] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:32.966785 4896 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:32.967279 4896 ts_tablet_manager.cc:1367] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 21:39:32.971347 4899 ts_tablet_manager.cc:1398] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:39:32.972530 4899 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:32.971916 4896 raft_consensus.cc:348] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.974296 4896 raft_consensus.cc:374] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:32.974613 4896 raft_consensus.cc:729] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:32.975342 4896 consensus_queue.cc:260] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:32.977655 4896 ts_tablet_manager.cc:1398] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.010s user 0.000s sys 0.007s
I20240502 21:39:32.978685 4896 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:32.979425 4899 tablet_bootstrap.cc:654] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.985373 4896 tablet_bootstrap.cc:654] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:32.991298 4899 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:32.991897 4899 ts_tablet_manager.cc:1367] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
I20240502 21:39:32.994853 4977 raft_consensus.cc:482] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:32.995535 4977 raft_consensus.cc:504] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.000797 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.001969 4776 raft_consensus.cc:2379] T a4d0c2f1f80b4015b25c866ee3d0c845 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:33.004941 4977 leader_election.cc:288] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.006363 4899 raft_consensus.cc:348] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.007532 4899 raft_consensus.cc:374] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.007961 4899 raft_consensus.cc:729] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.008935 4899 consensus_queue.cc:260] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.009246 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4d0c2f1f80b4015b25c866ee3d0c845" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:33.011505 4662 consensus_peers.cc:485] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Couldn't send request to peer df8e5a877b6640d692917083c9fb5282. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:33.019030 4899 ts_tablet_manager.cc:1398] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20240502 21:39:33.019212 4997 raft_consensus.cc:482] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.020697 4997 raft_consensus.cc:504] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.021019 4662 leader_election.cc:302] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.022253 4977 raft_consensus.cc:3037] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.022800 4997 leader_election.cc:288] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.024960 4896 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.025435 4896 ts_tablet_manager.cc:1367] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.047s user 0.018s sys 0.016s
I20240502 21:39:33.028970 4899 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.029623 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.027981 4896 raft_consensus.cc:348] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.030373 4896 raft_consensus.cc:374] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.030652 4896 raft_consensus.cc:729] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.031267 4896 consensus_queue.cc:260] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.031890 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e904cd5cd3b848d3b8dd89329851ba37" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.032788 4849 raft_consensus.cc:2379] T e904cd5cd3b848d3b8dd89329851ba37 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:33.033499 4896 ts_tablet_manager.cc:1398] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:39:33.034000 4737 leader_election.cc:302] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.034420 4896 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.035126 4997 raft_consensus.cc:3037] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.042330 4899 tablet_bootstrap.cc:654] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.042754 4997 raft_consensus.cc:2726] T e904cd5cd3b848d3b8dd89329851ba37 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.043548 4903 raft_consensus.cc:482] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.044453 4903 raft_consensus.cc:504] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.046658 4903 leader_election.cc:288] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.048261 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:33.049866 4740 leader_election.cc:341] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.051934 4896 tablet_bootstrap.cc:654] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.051847 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.052942 4899 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
W20240502 21:39:33.053275 4737 leader_election.cc:341] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.053414 4899 ts_tablet_manager.cc:1367] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20240502 21:39:33.053846 4895 tablet_bootstrap.cc:492] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.053790 4737 leader_election.cc:302] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.054425 4895 ts_tablet_manager.cc:1367] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.131s user 0.051s sys 0.051s
I20240502 21:39:33.056455 4899 raft_consensus.cc:348] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.056846 4895 raft_consensus.cc:348] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.057544 4895 raft_consensus.cc:374] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.057562 4899 raft_consensus.cc:374] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.057924 4895 raft_consensus.cc:729] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.058310 4899 raft_consensus.cc:729] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.058705 4903 raft_consensus.cc:2726] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.059155 4895 consensus_queue.cc:260] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.059094 4899 consensus_queue.cc:260] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.043792 4977 raft_consensus.cc:2726] T a4d0c2f1f80b4015b25c866ee3d0c845 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.064754 4895 ts_tablet_manager.cc:1398] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:39:33.065801 4895 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.070470 4899 ts_tablet_manager.cc:1398] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240502 21:39:33.071617 4895 tablet_bootstrap.cc:654] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.075053 4899 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.078002 4895 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.078580 4895 ts_tablet_manager.cc:1367] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20240502 21:39:33.081322 4895 raft_consensus.cc:348] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.082077 4895 raft_consensus.cc:374] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.082334 4895 raft_consensus.cc:729] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.082791 4899 tablet_bootstrap.cc:654] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.083057 4895 consensus_queue.cc:260] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.085418 4895 ts_tablet_manager.cc:1398] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240502 21:39:33.086467 4895 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.086927 4896 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.087401 4896 ts_tablet_manager.cc:1367] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.012s
I20240502 21:39:33.090090 4896 raft_consensus.cc:348] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.091001 4896 raft_consensus.cc:374] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.091217 4895 tablet_bootstrap.cc:654] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.091286 4977 raft_consensus.cc:482] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.091361 4896 raft_consensus.cc:729] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.091938 4977 raft_consensus.cc:504] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.092439 4896 consensus_queue.cc:260] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.094098 4977 leader_election.cc:288] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.095544 4896 ts_tablet_manager.cc:1398] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:39:33.096133 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.096776 4896 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.098834 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.099359 4899 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.099864 4899 ts_tablet_manager.cc:1367] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
W20240502 21:39:33.100858 4664 leader_election.cc:341] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:33.101747 4662 leader_election.cc:341] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.102177 4662 leader_election.cc:302] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.102418 4899 raft_consensus.cc:348] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.103112 4899 raft_consensus.cc:374] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.103269 4896 tablet_bootstrap.cc:654] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.103417 4899 raft_consensus.cc:729] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.103539 4977 raft_consensus.cc:2726] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.104370 4899 consensus_queue.cc:260] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.106645 4899 ts_tablet_manager.cc:1398] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240502 21:39:33.115188 4899 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.123571 4909 consensus_queue.cc:1035] T ba7a6cb1d5034911923934eee0d68132 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.127408 4951 raft_consensus.cc:482] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.128297 4899 tablet_bootstrap.cc:654] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.128003 4951 raft_consensus.cc:504] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.130486 4951 leader_election.cc:288] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:33.131377 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.132052 4895 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.132472 4896 tablet_bootstrap.cc:492] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.132968 4896 ts_tablet_manager.cc:1367] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.016s
I20240502 21:39:33.132501 4895 ts_tablet_manager.cc:1367] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.011s
I20240502 21:39:33.133781 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:33.135423 4817 leader_election.cc:341] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:33.136610 4818 leader_election.cc:341] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.136325 4896 raft_consensus.cc:348] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.137040 4818 leader_election.cc:302] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:33.137295 4896 raft_consensus.cc:374] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.137698 4896 raft_consensus.cc:729] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.137852 4908 raft_consensus.cc:2726] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.138562 4896 consensus_queue.cc:260] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.141024 4849 raft_consensus.cc:3037] T 8a8321179b7548c08b34565f04746387 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.141412 4896 ts_tablet_manager.cc:1398] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:39:33.142612 4896 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.148494 4774 raft_consensus.cc:3037] T 071379d3b0d24e4e9cfcb3f4cb2670fe P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.137550 4895 raft_consensus.cc:348] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.157543 4895 raft_consensus.cc:374] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.157968 4895 raft_consensus.cc:729] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.158694 4895 consensus_queue.cc:260] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.160553 4951 raft_consensus.cc:482] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.161013 4951 raft_consensus.cc:504] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.164444 4899 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.164939 4899 ts_tablet_manager.cc:1367] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.050s user 0.029s sys 0.007s
I20240502 21:39:33.166221 4951 leader_election.cc:288] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:33.166117 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.167373 4899 raft_consensus.cc:348] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.168139 4899 raft_consensus.cc:374] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.168449 4899 raft_consensus.cc:729] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.165091 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7a6cb1d5034911923934eee0d68132" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.169790 4896 tablet_bootstrap.cc:654] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.170456 4899 consensus_queue.cc:260] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.171387 4817 leader_election.cc:302] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:33.172833 4899 ts_tablet_manager.cc:1398] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:39:33.173091 4908 raft_consensus.cc:2726] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.173808 4899 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.176138 4908 consensus_queue.cc:1035] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.181625 4979 raft_consensus.cc:482] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.182200 4979 raft_consensus.cc:504] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.183027 4899 tablet_bootstrap.cc:654] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.185554 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.185876 4979 leader_election.cc:288] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.187042 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:33.187242 4662 leader_election.cc:341] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.187721 4772 raft_consensus.cc:2454] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.188825 4664 leader_election.cc:302] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.189978 4979 raft_consensus.cc:2781] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.190315 4979 raft_consensus.cc:482] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.190621 4979 raft_consensus.cc:3037] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.192720 4975 raft_consensus.cc:482] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.193200 4975 raft_consensus.cc:504] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.195458 4975 leader_election.cc:288] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.196591 4899 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.197315 4899 ts_tablet_manager.cc:1367] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.015s
I20240502 21:39:33.197496 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.198330 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.199154 4773 raft_consensus.cc:2454] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.200700 4664 leader_election.cc:302] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:33.201572 4975 raft_consensus.cc:2781] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.201948 4975 raft_consensus.cc:482] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.202368 4975 raft_consensus.cc:3037] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:33.203457 4662 leader_election.cc:341] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.205129 4932 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.205714 4932 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.207543 4932 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.208577 4896 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.209134 4896 ts_tablet_manager.cc:1367] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.067s user 0.020s sys 0.006s
I20240502 21:39:33.211742 4896 raft_consensus.cc:348] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.211923 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.212639 4896 raft_consensus.cc:374] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.213004 4896 raft_consensus.cc:729] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.213120 4849 raft_consensus.cc:2379] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:33.213935 4975 raft_consensus.cc:504] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.213786 4896 consensus_queue.cc:260] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.216645 4896 ts_tablet_manager.cc:1398] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:39:33.217712 4896 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.219699 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.219964 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.220783 4774 raft_consensus.cc:2379] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
W20240502 21:39:33.221107 4662 leader_election.cc:341] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.222110 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73bd5217e94b49bab1732f9fd0c8d79c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.222702 4773 raft_consensus.cc:3037] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.223744 4899 raft_consensus.cc:348] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.224468 4899 raft_consensus.cc:374] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.224728 4899 raft_consensus.cc:729] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.225440 4664 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.225351 4899 consensus_queue.cc:260] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.225786 4895 ts_tablet_manager.cc:1398] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.092s user 0.005s sys 0.002s
I20240502 21:39:33.226318 4932 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.227362 4899 ts_tablet_manager.cc:1398] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20240502 21:39:33.228293 4899 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.230484 4895 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.234025 4899 tablet_bootstrap.cc:654] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.238093 4773 raft_consensus.cc:2454] T 73bd5217e94b49bab1732f9fd0c8d79c P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.240898 4932 raft_consensus.cc:2726] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.243070 4979 raft_consensus.cc:504] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.251546 4895 tablet_bootstrap.cc:654] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.252420 4975 leader_election.cc:288] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.253746 4979 leader_election.cc:288] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.250722 4896 tablet_bootstrap.cc:654] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.255645 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
W20240502 21:39:33.258891 4662 leader_election.cc:341] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.262521 4899 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.262995 4899 ts_tablet_manager.cc:1367] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.035s user 0.019s sys 0.003s
I20240502 21:39:33.265503 4899 raft_consensus.cc:348] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.266223 4899 raft_consensus.cc:374] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.266495 4899 raft_consensus.cc:729] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.267406 4896 tablet_bootstrap.cc:492] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.268224 4896 ts_tablet_manager.cc:1367] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.005s
I20240502 21:39:33.269177 4979 raft_consensus.cc:482] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.267144 4899 consensus_queue.cc:260] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.269680 4979 raft_consensus.cc:504] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.270002 4664 leader_election.cc:302] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.274415 4975 raft_consensus.cc:2781] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.274855 4979 leader_election.cc:288] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.275426 4895 tablet_bootstrap.cc:492] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.272157 4899 ts_tablet_manager.cc:1398] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240502 21:39:33.275956 4895 ts_tablet_manager.cc:1367] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.012s
I20240502 21:39:33.271129 4896 raft_consensus.cc:348] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.276690 4896 raft_consensus.cc:374] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.274900 4975 raft_consensus.cc:686] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:33.277513 4997 raft_consensus.cc:482] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.277824 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.278015 4997 raft_consensus.cc:504] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.278568 4849 raft_consensus.cc:2454] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.276999 4896 raft_consensus.cc:729] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.277834 4975 consensus_queue.cc:237] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.280368 4899 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.279932 4896 consensus_queue.cc:260] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.280820 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.281400 4662 leader_election.cc:302] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:33.280027 4997 leader_election.cc:288] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.282251 4979 raft_consensus.cc:2781] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.282475 4896 ts_tablet_manager.cc:1398] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 21:39:33.282632 4979 raft_consensus.cc:482] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.282989 4979 raft_consensus.cc:3037] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.283401 4896 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.283994 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:33.285523 4664 consensus_peers.cc:485] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Couldn't send request to peer ddea410051614f23b93c7342ec7cc545. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:39:33.286525 4737 leader_election.cc:341] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.286406 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:33.287074 4664 leader_election.cc:341] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:33.288121 4740 leader_election.cc:341] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.288372 4932 raft_consensus.cc:482] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.288609 4740 leader_election.cc:302] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.288898 4932 raft_consensus.cc:504] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.289947 4997 raft_consensus.cc:2726] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.290684 4899 tablet_bootstrap.cc:654] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.269349 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.291261 4773 raft_consensus.cc:3037] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.287786 4951 consensus_queue.cc:1035] T a4d0c2f1f80b4015b25c866ee3d0c845 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 21:39:33.292145 4932 leader_election.cc:288] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.297628 4772 raft_consensus.cc:1261] T 2b5e2e41273949e2afa8bc07812c9ca7 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:33.298887 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.299605 4849 raft_consensus.cc:2454] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.301445 4662 leader_election.cc:302] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:33.302310 4932 raft_consensus.cc:2781] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.302676 4932 raft_consensus.cc:482] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.303010 4932 raft_consensus.cc:3037] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.306749 4773 raft_consensus.cc:2454] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.307037 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.311034 4772 raft_consensus.cc:2454] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.307698 4951 consensus_queue.cc:1035] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:33.308621 4997 raft_consensus.cc:482] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.309137 4979 raft_consensus.cc:504] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.316108 4997 raft_consensus.cc:504] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.322400 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.323302 4849 raft_consensus.cc:3037] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.309763 4895 raft_consensus.cc:348] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.292161 4896 tablet_bootstrap.cc:654] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.324515 4895 raft_consensus.cc:374] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.324854 4895 raft_consensus.cc:729] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.325516 4895 consensus_queue.cc:260] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.327855 4895 ts_tablet_manager.cc:1398] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.051s user 0.002s sys 0.002s
I20240502 21:39:33.316433 4979 leader_election.cc:288] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.328457 4699 raft_consensus.cc:1261] T 2b5e2e41273949e2afa8bc07812c9ca7 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 21:39:33.328894 4997 leader_election.cc:288] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.330536 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.331188 4852 raft_consensus.cc:2454] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:33.332476 4737 leader_election.cc:302] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:33.333689 4932 raft_consensus.cc:504] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.328192 4593 catalog_manager.cc:5484] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.335029 4908 consensus_queue.cc:1035] T 2b5e2e41273949e2afa8bc07812c9ca7 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:39:33.338513 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:33.338979 4997 raft_consensus.cc:2781] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.339450 4997 raft_consensus.cc:482] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.343420 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.344059 4774 raft_consensus.cc:3037] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.344743 4849 raft_consensus.cc:2454] T 63db76a96b4c474994164c8a7086bbbd P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.343755 4997 raft_consensus.cc:3037] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.348688 4979 consensus_queue.cc:1035] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.350430 4662 leader_election.cc:302] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:33.351244 4979 raft_consensus.cc:2781] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.351704 4979 raft_consensus.cc:686] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:33.352559 4979 consensus_queue.cc:237] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.356810 4774 raft_consensus.cc:2454] T 30ff5dbe7549485e96acef1baa80cd02 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.359537 4899 tablet_bootstrap.cc:492] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.360075 4899 ts_tablet_manager.cc:1367] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.017s
I20240502 21:39:33.369501 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ff5dbe7549485e96acef1baa80cd02" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.377777 4849 raft_consensus.cc:3037] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.341619 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.340101 4895 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.340639 4932 leader_election.cc:288] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.377908 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.380681 4697 raft_consensus.cc:2454] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:33.374503 4899 raft_consensus.cc:348] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.383599 4899 raft_consensus.cc:374] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.385195 4899 raft_consensus.cc:729] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.385483 4975 consensus_queue.cc:1035] T e904cd5cd3b848d3b8dd89329851ba37 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.386015 4899 consensus_queue.cc:260] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.387588 4664 leader_election.cc:302] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.389545 4899 ts_tablet_manager.cc:1398] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.029s user 0.003s sys 0.009s
I20240502 21:39:33.393621 4899 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.394040 4932 raft_consensus.cc:2781] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.394807 4932 raft_consensus.cc:686] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:33.373831 4997 raft_consensus.cc:504] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.397214 4932 consensus_queue.cc:237] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.398185 4664 leader_election.cc:302] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
W20240502 21:39:33.410142 4664 leader_election.cc:341] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.410071 4979 raft_consensus.cc:2781] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.411021 4979 raft_consensus.cc:686] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:33.412191 4979 consensus_queue.cc:237] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.414762 4849 raft_consensus.cc:2454] T 30ff5dbe7549485e96acef1baa80cd02 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.443214 4997 leader_election.cc:288] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.443866 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:33.456512 4699 raft_consensus.cc:3037] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.444290 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e03d7dc5734e2e9e52437b5a1bc124" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.457793 4852 raft_consensus.cc:3037] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.446899 4896 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.463089 4896 ts_tablet_manager.cc:1367] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.180s user 0.005s sys 0.020s
I20240502 21:39:33.463594 4699 raft_consensus.cc:2454] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:33.447930 4849 raft_consensus.cc:3037] T ba7a6cb1d5034911923934eee0d68132 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.465440 4740 leader_election.cc:302] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:33.466490 4997 raft_consensus.cc:2781] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.466986 4997 raft_consensus.cc:686] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:33.451325 4593 catalog_manager.cc:5484] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.453889 4993 raft_consensus.cc:482] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.467787 4997 consensus_queue.cc:237] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.465873 4896 raft_consensus.cc:348] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.468551 4993 raft_consensus.cc:504] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.469054 4896 raft_consensus.cc:374] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.470552 4993 leader_election.cc:288] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.469499 4896 raft_consensus.cc:729] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.474393 4899 tablet_bootstrap.cc:654] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.474129 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.474504 4896 consensus_queue.cc:260] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.475327 4850 raft_consensus.cc:2379] T 21fbea6bad03441f8109d77239412009 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:33.476799 4896 ts_tablet_manager.cc:1398] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:39:33.477715 4896 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.483675 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21fbea6bad03441f8109d77239412009" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.479130 4979 consensus_queue.cc:1035] T 21fbea6bad03441f8109d77239412009 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240502 21:39:33.486158 4740 leader_election.cc:302] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.487856 4993 raft_consensus.cc:3037] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.491201 4896 tablet_bootstrap.cc:654] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.477743 4895 tablet_bootstrap.cc:654] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.506785 4899 tablet_bootstrap.cc:492] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.507293 4899 ts_tablet_manager.cc:1367] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.005s
I20240502 21:39:33.510227 4899 raft_consensus.cc:348] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.511040 4899 raft_consensus.cc:374] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.511324 4899 raft_consensus.cc:729] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.511960 4899 consensus_queue.cc:260] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.512614 4852 raft_consensus.cc:2454] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:33.514297 4899 ts_tablet_manager.cc:1398] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 21:39:33.515192 4899 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.521127 4899 tablet_bootstrap.cc:654] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.525291 4591 catalog_manager.cc:5484] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.527321 4591 catalog_manager.cc:5484] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.527108 4896 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.529531 4896 ts_tablet_manager.cc:1367] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20240502 21:39:33.533022 4899 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.533604 4899 ts_tablet_manager.cc:1367] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20240502 21:39:33.533694 4895 tablet_bootstrap.cc:492] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.534142 4895 ts_tablet_manager.cc:1367] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.194s user 0.012s sys 0.001s
I20240502 21:39:33.532270 4896 raft_consensus.cc:348] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.536713 4896 raft_consensus.cc:374] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.536971 4896 raft_consensus.cc:729] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.537351 4899 raft_consensus.cc:348] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.537669 4896 consensus_queue.cc:260] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.538031 4899 raft_consensus.cc:374] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.536585 4895 raft_consensus.cc:348] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.538452 4899 raft_consensus.cc:729] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.538728 4895 raft_consensus.cc:374] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.539019 4895 raft_consensus.cc:729] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.539186 4899 consensus_queue.cc:260] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.539716 4979 raft_consensus.cc:482] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.538034 4993 raft_consensus.cc:2726] T 21fbea6bad03441f8109d77239412009 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.540252 4979 raft_consensus.cc:504] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.541400 4899 ts_tablet_manager.cc:1398] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 21:39:33.542153 4896 ts_tablet_manager.cc:1398] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20240502 21:39:33.542369 4899 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.543061 4896 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.542282 4979 leader_election.cc:288] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.543130 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.543882 4772 raft_consensus.cc:2454] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.543846 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.539737 4895 consensus_queue.cc:260] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.545326 4852 raft_consensus.cc:2454] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.546396 4662 leader_election.cc:302] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:33.546886 4895 ts_tablet_manager.cc:1398] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20240502 21:39:33.547215 4979 raft_consensus.cc:2781] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.547547 4979 raft_consensus.cc:482] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.547775 4895 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.547870 4979 raft_consensus.cc:3037] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.548594 4899 tablet_bootstrap.cc:654] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.553260 4895 tablet_bootstrap.cc:654] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.555097 4979 raft_consensus.cc:504] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.560153 4899 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.560781 4899 ts_tablet_manager.cc:1367] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240502 21:39:33.563198 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.563867 4772 raft_consensus.cc:3037] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.563791 4899 raft_consensus.cc:348] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.564229 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe02980826846ccaa247803f60d8f0c" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.565114 4852 raft_consensus.cc:3037] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.565346 4979 leader_election.cc:288] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.564617 4899 raft_consensus.cc:374] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.566017 4899 raft_consensus.cc:729] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.567183 4896 tablet_bootstrap.cc:654] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.566802 4899 consensus_queue.cc:260] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.570020 4895 tablet_bootstrap.cc:492] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.570493 4895 ts_tablet_manager.cc:1367] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20240502 21:39:33.573025 4895 raft_consensus.cc:348] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.573590 4772 raft_consensus.cc:2454] T dfe02980826846ccaa247803f60d8f0c P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.574591 4899 ts_tablet_manager.cc:1398] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240502 21:39:33.575059 4664 leader_election.cc:302] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:33.575536 4899 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.575919 4979 raft_consensus.cc:2781] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.576429 4979 raft_consensus.cc:686] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:33.577263 4979 consensus_queue.cc:237] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.575582 4593 catalog_manager.cc:5484] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.581530 4899 tablet_bootstrap.cc:654] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.573551 4852 raft_consensus.cc:2454] T dfe02980826846ccaa247803f60d8f0c P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:33.573768 4895 raft_consensus.cc:374] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.593515 4895 raft_consensus.cc:729] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.594538 4895 consensus_queue.cc:260] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.598143 4895 ts_tablet_manager.cc:1398] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20240502 21:39:33.597774 4899 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.599594 4899 ts_tablet_manager.cc:1367] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20240502 21:39:33.601601 4896 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.602057 4896 ts_tablet_manager.cc:1367] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.026s
I20240502 21:39:33.602113 4899 raft_consensus.cc:348] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.602895 4899 raft_consensus.cc:374] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.603157 4899 raft_consensus.cc:729] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.603865 4899 consensus_queue.cc:260] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.604679 4896 raft_consensus.cc:348] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.605998 4896 raft_consensus.cc:374] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.606235 4899 ts_tablet_manager.cc:1398] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:39:33.607141 4899 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.608036 4895 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.612457 4899 tablet_bootstrap.cc:654] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.613629 4895 tablet_bootstrap.cc:654] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.606243 4896 raft_consensus.cc:729] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.615810 4896 consensus_queue.cc:260] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.618494 4896 ts_tablet_manager.cc:1398] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:39:33.619666 4896 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.621609 4593 catalog_manager.cc:5484] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.627120 4896 tablet_bootstrap.cc:654] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.629968 4895 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.630615 4895 ts_tablet_manager.cc:1367] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.008s
I20240502 21:39:33.630651 4899 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.631307 4899 ts_tablet_manager.cc:1367] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.009s
I20240502 21:39:33.633263 4895 raft_consensus.cc:348] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.634001 4895 raft_consensus.cc:374] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.634415 4895 raft_consensus.cc:729] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.633860 4899 raft_consensus.cc:348] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.634899 4899 raft_consensus.cc:374] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.635344 4899 raft_consensus.cc:729] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.635169 4895 consensus_queue.cc:260] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.637576 4895 ts_tablet_manager.cc:1398] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 21:39:33.639164 4895 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.646103 4895 tablet_bootstrap.cc:654] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.639528 4899 consensus_queue.cc:260] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.647085 4997 raft_consensus.cc:482] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.647343 4899 ts_tablet_manager.cc:1398] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240502 21:39:33.647569 4997 raft_consensus.cc:504] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.648494 4899 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.649663 4997 leader_election.cc:288] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.652755 4896 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.652591 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.653189 4896 ts_tablet_manager.cc:1367] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.000s
I20240502 21:39:33.653246 4993 raft_consensus.cc:482] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.653777 4993 raft_consensus.cc:504] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:33.654101 4740 leader_election.cc:341] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.655129 4903 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.655606 4903 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.656888 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.657534 4699 raft_consensus.cc:2454] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:33.657994 4903 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.657869 4896 raft_consensus.cc:348] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.658591 4896 raft_consensus.cc:374] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.658825 4993 leader_election.cc:288] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.658862 4896 raft_consensus.cc:729] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.661147 4740 leader_election.cc:302] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:33.661628 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.661908 4899 tablet_bootstrap.cc:654] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.662142 4993 raft_consensus.cc:2781] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.662530 4993 raft_consensus.cc:482] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.662871 4993 raft_consensus.cc:3037] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:33.663489 4740 leader_election.cc:341] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.664206 4896 consensus_queue.cc:260] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.666990 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.667961 4908 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.669813 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.670768 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.671759 4908 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.673789 4908 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:33.675441 4896 ts_tablet_manager.cc:1398] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20240502 21:39:33.675580 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:33.676890 4818 leader_election.cc:341] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.677703 4849 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:33.680256 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.680918 4772 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:33.682158 4817 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
W20240502 21:39:33.684201 4737 leader_election.cc:341] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.684835 4896 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.684707 4737 leader_election.cc:302] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.686430 4997 raft_consensus.cc:2726] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:39:33.687898 4737 leader_election.cc:341] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.689883 4737 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:33.691277 4997 raft_consensus.cc:2781] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.691637 4997 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.691947 4997 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.693241 4896 tablet_bootstrap.cc:654] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.690589 4908 raft_consensus.cc:2781] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.695226 4908 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.695583 4908 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.699231 4895 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.699901 4895 ts_tablet_manager.cc:1367] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.021s
I20240502 21:39:33.703481 4895 raft_consensus.cc:348] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.704504 4895 raft_consensus.cc:374] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.704840 4895 raft_consensus.cc:729] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.707362 4908 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.707679 4895 consensus_queue.cc:260] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.714412 4895 ts_tablet_manager.cc:1398] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20240502 21:39:33.714946 4997 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.715648 4895 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.717139 4997 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.718751 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:33.720418 4740 leader_election.cc:341] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.721506 4934 consensus_queue.cc:1035] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.726559 4849 raft_consensus.cc:3037] T 73bd5217e94b49bab1732f9fd0c8d79c P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:33.736757 4737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 46945.3us
I20240502 21:39:33.738554 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.739389 4851 raft_consensus.cc:2379] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:33.739573 4990 raft_consensus.cc:482] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.740031 4990 raft_consensus.cc:504] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.741998 4990 leader_election.cc:288] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:33.742194 4737 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.743870 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.744277 4997 raft_consensus.cc:2726] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.745592 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515491e619e145a292c5e5f654cc63ee" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.746723 4776 raft_consensus.cc:2379] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:33.747890 4934 consensus_queue.cc:1035] T 73bd5217e94b49bab1732f9fd0c8d79c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.749145 4817 leader_election.cc:302] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:33.752151 4947 raft_consensus.cc:3037] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.754374 4993 raft_consensus.cc:504] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.759994 4993 leader_election.cc:288] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.762601 4947 raft_consensus.cc:2726] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.762722 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:33.763825 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:33.763842 4699 raft_consensus.cc:3037] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.771482 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
W20240502 21:39:33.776757 4818 leader_election.cc:341] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.783191 4776 raft_consensus.cc:3037] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.786592 4934 consensus_queue.cc:1035] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.790038 4979 raft_consensus.cc:482] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.790524 4979 raft_consensus.cc:504] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:33.791366 4737 leader_election.cc:341] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.792753 4979 leader_election.cc:288] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.794684 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.795787 4997 raft_consensus.cc:482] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.796458 4979 consensus_queue.cc:1035] T 63db76a96b4c474994164c8a7086bbbd P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:33.796478 4997 raft_consensus.cc:504] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:33.798714 4664 leader_election.cc:341] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.802249 4934 consensus_queue.cc:1035] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:33.804086 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.772012 4699 raft_consensus.cc:2454] T f05bc1efec5f4b95a68f7b46fdd719ad P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:33.804786 4850 raft_consensus.cc:2454] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.807125 4740 leader_election.cc:302] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.808229 4903 raft_consensus.cc:2781] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:33.808888 4903 raft_consensus.cc:686] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:33.809998 4903 consensus_queue.cc:237] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.811062 4662 leader_election.cc:302] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:33.813393 4979 raft_consensus.cc:2781] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.813786 4979 raft_consensus.cc:482] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.814118 4979 raft_consensus.cc:3037] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.839612 4908 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:33.840648 5001 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.841150 5001 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.841518 4997 leader_election.cc:288] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.841899 4895 tablet_bootstrap.cc:654] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.845589 4896 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:33.845738 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.846294 4896 ts_tablet_manager.cc:1367] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.162s user 0.018s sys 0.002s
I20240502 21:39:33.848460 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.849359 4775 raft_consensus.cc:2454] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.855247 4899 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:33.855737 4899 ts_tablet_manager.cc:1367] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.208s user 0.058s sys 0.102s
I20240502 21:39:33.834846 4934 consensus_queue.cc:1035] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20240502 21:39:33.856717 4896 raft_consensus.cc:348] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.858070 4896 raft_consensus.cc:374] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.858387 4896 raft_consensus.cc:729] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.859025 4664 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:33.859112 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.859438 4896 consensus_queue.cc:260] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.859680 5001 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.859611 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63db76a96b4c474994164c8a7086bbbd" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.859479 4932 consensus_queue.cc:1035] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 21:39:33.870525 5001 raft_consensus.cc:2781] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:33.872718 5001 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:33.873103 5001 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.869555 4975 raft_consensus.cc:482] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.877348 4975 raft_consensus.cc:504] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.875398 4775 raft_consensus.cc:2379] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:33.881632 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.875977 4899 raft_consensus.cc:348] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.882846 4899 raft_consensus.cc:374] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.883224 4899 raft_consensus.cc:729] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.887748 4899 consensus_queue.cc:260] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.890187 4817 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:33.891121 4951 raft_consensus.cc:2726] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:33.896039 4979 raft_consensus.cc:504] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.898743 4979 leader_election.cc:288] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.899703 4975 leader_election.cc:288] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.905019 4899 ts_tablet_manager.cc:1398] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.049s user 0.009s sys 0.000s
I20240502 21:39:33.906093 4899 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:33.903129 5001 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.911103 4896 ts_tablet_manager.cc:1398] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.064s user 0.008s sys 0.005s
I20240502 21:39:33.912431 4896 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:33.936991 5001 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.937150 4899 tablet_bootstrap.cc:654] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.938354 4895 tablet_bootstrap.cc:492] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:33.938884 4895 ts_tablet_manager.cc:1367] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.223s user 0.017s sys 0.009s
I20240502 21:39:33.941462 4895 raft_consensus.cc:348] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.941991 4895 raft_consensus.cc:374] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:33.942206 4895 raft_consensus.cc:729] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:33.942747 4895 consensus_queue.cc:260] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.943873 4895 ts_tablet_manager.cc:1398] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240502 21:39:33.927346 4662 leader_election.cc:341] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.945806 5068 raft_consensus.cc:482] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:39:33.934345 4664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 75478.7us
I20240502 21:39:33.945873 4903 raft_consensus.cc:482] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.950039 4903 raft_consensus.cc:504] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.949410 5068 raft_consensus.cc:504] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.953009 5068 leader_election.cc:288] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.952913 4593 catalog_manager.cc:5484] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:33.958375 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:33.959329 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.959607 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.952189 4903 leader_election.cc:288] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.961990 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:33.962725 4774 raft_consensus.cc:2454] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:33.965400 4896 tablet_bootstrap.cc:654] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:33.972679 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.973443 4848 raft_consensus.cc:2454] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:33.965476 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.974220 5068 raft_consensus.cc:482] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:33.965679 4740 leader_election.cc:302] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:33.966077 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:33.966276 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
W20240502 21:39:33.966820 4662 leader_election.cc:341] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:33.967833 4895 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:33.968326 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:33.988173 4903 raft_consensus.cc:2726] T 63db76a96b4c474994164c8a7086bbbd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:39:33.985585 4850 raft_consensus.cc:3037] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.974720 5068 raft_consensus.cc:504] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:33.989027 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:33.991194 4774 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:33.995772 5068 leader_election.cc:288] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:33.999431 4774 raft_consensus.cc:2454] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
W20240502 21:39:33.999665 4664 leader_election.cc:341] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.000442 4664 leader_election.cc:302] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:34.002106 5071 raft_consensus.cc:2781] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.002507 5071 raft_consensus.cc:482] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.002846 5071 raft_consensus.cc:3037] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.002908 5001 consensus_queue.cc:1035] T 30ff5dbe7549485e96acef1baa80cd02 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.004387 4737 leader_election.cc:302] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
W20240502 21:39:34.005443 4662 leader_election.cc:341] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.005776 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.006851 4903 raft_consensus.cc:2781] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.007228 4903 raft_consensus.cc:482] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.007572 4903 raft_consensus.cc:3037] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.007643 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:34.009434 4737 leader_election.cc:341] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:34.010251 4737 leader_election.cc:341] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:34.017014 4740 leader_election.cc:341] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.027792 4903 raft_consensus.cc:504] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.024230 4740 leader_election.cc:302] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
W20240502 21:39:34.046659 4740 leader_election.cc:341] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.049640 4997 raft_consensus.cc:2726] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.050601 5068 consensus_queue.cc:1035] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 21:39:34.054244 4850 raft_consensus.cc:2454] T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:34.055414 4903 leader_election.cc:288] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:34.048013 4740 leader_election.cc:341] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.057260 4740 leader_election.cc:302] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.058701 4664 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.059497 5068 raft_consensus.cc:2726] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.061144 5001 raft_consensus.cc:2781] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.061780 5001 raft_consensus.cc:686] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:34.031750 5071 raft_consensus.cc:504] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.062459 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:34.062641 5001 consensus_queue.cc:237] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.058213 4662 leader_election.cc:302] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:34.068418 4979 raft_consensus.cc:2781] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.069010 4979 raft_consensus.cc:686] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:34.070037 4979 consensus_queue.cc:237] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.071727 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:34.070029 5071 leader_election.cc:288] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.072423 4852 raft_consensus.cc:3037] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.081596 4899 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:34.082283 4899 ts_tablet_manager.cc:1367] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.176s user 0.021s sys 0.000s
I20240502 21:39:34.082440 4895 tablet_bootstrap.cc:654] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.088915 4896 tablet_bootstrap.cc:492] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:34.089637 4896 ts_tablet_manager.cc:1367] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.177s user 0.010s sys 0.008s
I20240502 21:39:34.090940 4899 raft_consensus.cc:348] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.091382 4852 raft_consensus.cc:2454] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:34.091895 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:34.093741 4899 raft_consensus.cc:374] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.093135 4896 raft_consensus.cc:348] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.094691 4899 raft_consensus.cc:729] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.095141 4896 raft_consensus.cc:374] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.095469 4896 raft_consensus.cc:729] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.095849 4899 consensus_queue.cc:260] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.097862 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.098456 4772 raft_consensus.cc:3037] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:34.098593 4740 leader_election.cc:341] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.100191 4899 ts_tablet_manager.cc:1398] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 21:39:34.101117 4899 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:34.102293 5068 consensus_queue.cc:1035] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.102461 5001 consensus_queue.cc:1035] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:34.096783 4896 consensus_queue.cc:260] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:34.098446 4662 leader_election.cc:341] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.118952 4772 raft_consensus.cc:2454] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:34.109748 4737 leader_election.cc:302] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:34.122241 4993 raft_consensus.cc:2781] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.123121 4993 raft_consensus.cc:686] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:34.121309 4591 catalog_manager.cc:5484] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:34.124521 4896 ts_tablet_manager.cc:1398] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.034s user 0.009s sys 0.000s
I20240502 21:39:34.124001 4993 consensus_queue.cc:237] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.126118 4896 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:34.129938 4664 leader_election.cc:302] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.131670 5001 raft_consensus.cc:2781] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.134061 5001 raft_consensus.cc:686] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:34.135496 5001 consensus_queue.cc:237] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.138898 4896 tablet_bootstrap.cc:654] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.129472 4591 catalog_manager.cc:5484] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:34.139885 4899 tablet_bootstrap.cc:654] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.140842 4895 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.141316 4895 ts_tablet_manager.cc:1367] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.174s user 0.011s sys 0.001s
I20240502 21:39:34.157079 4979 consensus_queue.cc:1035] T dfe02980826846ccaa247803f60d8f0c P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:34.168766 4594 catalog_manager.cc:5484] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:34.199540 4896 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:34.200018 4896 ts_tablet_manager.cc:1367] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.074s user 0.017s sys 0.010s
I20240502 21:39:34.206589 4899 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:34.206360 4895 raft_consensus.cc:348] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.207096 4899 ts_tablet_manager.cc:1367] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.013s
I20240502 21:39:34.207361 4895 raft_consensus.cc:374] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.208549 4895 raft_consensus.cc:729] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.209389 4895 consensus_queue.cc:260] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.210001 4899 raft_consensus.cc:348] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.211025 4899 raft_consensus.cc:374] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.211382 4899 raft_consensus.cc:729] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.212352 4895 ts_tablet_manager.cc:1398] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.071s user 0.006s sys 0.007s
I20240502 21:39:34.213353 4895 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.224834 4895 tablet_bootstrap.cc:654] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.213459 4896 raft_consensus.cc:348] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.225224 4896 raft_consensus.cc:374] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.225538 4896 raft_consensus.cc:729] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.226179 4896 consensus_queue.cc:260] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.228246 4896 ts_tablet_manager.cc:1398] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20240502 21:39:34.229198 4896 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:34.212242 4899 consensus_queue.cc:260] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.232252 4899 ts_tablet_manager.cc:1398] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20240502 21:39:34.235114 4899 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:34.237906 4896 tablet_bootstrap.cc:654] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.240623 4591 catalog_manager.cc:5484] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:34.245319 4993 raft_consensus.cc:482] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.245944 4993 raft_consensus.cc:504] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.246868 4895 tablet_bootstrap.cc:492] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.247502 4895 ts_tablet_manager.cc:1367] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.007s
I20240502 21:39:34.248044 4899 tablet_bootstrap.cc:654] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.249779 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.250818 4895 raft_consensus.cc:348] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.252255 4895 raft_consensus.cc:374] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.248119 4993 leader_election.cc:288] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.252593 4895 raft_consensus.cc:729] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.253238 4895 consensus_queue.cc:260] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.255203 4895 ts_tablet_manager.cc:1398] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:39:34.256062 4895 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.257439 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:34.258292 4979 raft_consensus.cc:482] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.256199 4896 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:34.259018 4896 ts_tablet_manager.cc:1367] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.008s
W20240502 21:39:34.258905 4737 leader_election.cc:341] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.258776 4979 raft_consensus.cc:504] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.262364 4979 leader_election.cc:288] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.263355 4896 raft_consensus.cc:348] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.264147 4896 raft_consensus.cc:374] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.264470 4896 raft_consensus.cc:729] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
W20240502 21:39:34.265432 4740 leader_election.cc:341] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.265193 4896 consensus_queue.cc:260] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.267647 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:34.270103 4740 leader_election.cc:302] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.270987 4993 raft_consensus.cc:2726] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.272549 4895 tablet_bootstrap.cc:654] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.273195 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.273869 4776 raft_consensus.cc:2454] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:34.268194 4896 ts_tablet_manager.cc:1398] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 21:39:34.275472 4899 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:34.275964 4899 ts_tablet_manager.cc:1367] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.011s
I20240502 21:39:34.276019 4896 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
W20240502 21:39:34.276165 4662 leader_election.cc:341] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.278565 4899 raft_consensus.cc:348] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.279304 4899 raft_consensus.cc:374] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.279583 4899 raft_consensus.cc:729] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.280297 4899 consensus_queue.cc:260] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.281414 4896 tablet_bootstrap.cc:654] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.283066 4899 ts_tablet_manager.cc:1398] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240502 21:39:34.287874 4895 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.288380 4895 ts_tablet_manager.cc:1367] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.006s
I20240502 21:39:34.288478 4899 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:34.291338 4895 raft_consensus.cc:348] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.292110 4895 raft_consensus.cc:374] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.292441 4895 raft_consensus.cc:729] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.293167 4664 leader_election.cc:302] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.294772 4979 raft_consensus.cc:2781] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.295244 4979 raft_consensus.cc:482] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.295598 4899 tablet_bootstrap.cc:654] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.295686 4979 raft_consensus.cc:3037] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.305377 4979 raft_consensus.cc:504] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.293182 4895 consensus_queue.cc:260] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.310392 4895 ts_tablet_manager.cc:1398] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20240502 21:39:34.311652 4895 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.323338 4979 leader_election.cc:288] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.324396 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:34.324661 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.325183 4775 raft_consensus.cc:3037] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.325785 4979 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:39:34.326086 4662 leader_election.cc:341] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.326402 4979 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.332700 4979 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.333189 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:34.334080 4852 raft_consensus.cc:2379] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:34.336656 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.337466 4776 raft_consensus.cc:2379] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:34.339115 4664 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.340786 5073 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.350423 5073 raft_consensus.cc:2726] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.351943 4895 tablet_bootstrap.cc:654] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.364678 4895 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.365311 4895 ts_tablet_manager.cc:1367] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.002s
I20240502 21:39:34.369299 4895 raft_consensus.cc:348] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.370375 4895 raft_consensus.cc:374] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.370743 4895 raft_consensus.cc:729] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.372023 4895 consensus_queue.cc:260] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.376509 4895 ts_tablet_manager.cc:1398] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20240502 21:39:34.378266 4895 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.386034 4895 tablet_bootstrap.cc:654] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.392693 4775 raft_consensus.cc:2454] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:34.408819 4908 raft_consensus.cc:482] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.409524 4664 leader_election.cc:302] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.409580 4908 raft_consensus.cc:504] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.414916 4908 leader_election.cc:288] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.416736 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.425046 4993 consensus_queue.cc:1035] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:34.425863 4852 raft_consensus.cc:3037] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.412000 4979 raft_consensus.cc:2781] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.429284 5071 raft_consensus.cc:482] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.430011 5071 raft_consensus.cc:504] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.432564 4979 raft_consensus.cc:686] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:34.433871 4979 consensus_queue.cc:237] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.440637 5071 leader_election.cc:288] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.444792 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.399147 4895 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.402701 4951 raft_consensus.cc:482] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.446520 4951 raft_consensus.cc:504] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.449368 4951 leader_election.cc:288] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.450078 4951 raft_consensus.cc:482] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.451922 4951 raft_consensus.cc:504] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.455933 4895 ts_tablet_manager.cc:1367] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.078s user 0.016s sys 0.004s
I20240502 21:39:34.450501 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:34.458258 4818 leader_election.cc:341] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.455029 4951 leader_election.cc:288] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.465946 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238a1f48ddb14387bc2aa4419ff8879e" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:34.467824 4818 leader_election.cc:341] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.463722 4895 raft_consensus.cc:348] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.470836 4908 raft_consensus.cc:482] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.471554 4908 raft_consensus.cc:504] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.469110 4895 raft_consensus.cc:374] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.475461 4895 raft_consensus.cc:729] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.476153 4895 consensus_queue.cc:260] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.479146 4993 consensus_queue.cc:1035] T f05bc1efec5f4b95a68f7b46fdd719ad P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.482156 4908 leader_election.cc:288] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.483399 4993 raft_consensus.cc:482] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.484181 4993 raft_consensus.cc:504] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.492455 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:34.494515 4737 leader_election.cc:341] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.495304 4993 leader_election.cc:288] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.495618 4895 ts_tablet_manager.cc:1398] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20240502 21:39:34.497985 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.498137 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.499532 4699 raft_consensus.cc:2454] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:34.502557 4895 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.502630 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238a1f48ddb14387bc2aa4419ff8879e" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.516757 4772 raft_consensus.cc:2454] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:34.504140 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.518875 4776 raft_consensus.cc:2454] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:34.505602 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05bc1efec5f4b95a68f7b46fdd719ad" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.506951 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c35cadfff54236bbf05e32f945728f" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:34.521344 4850 raft_consensus.cc:2379] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:34.509980 4740 leader_election.cc:302] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.523039 4997 raft_consensus.cc:2781] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.523525 4997 raft_consensus.cc:482] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.524013 4997 raft_consensus.cc:3037] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.537604 5104 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.538365 5104 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.541110 5104 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.546010 4662 leader_election.cc:302] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.546334 4817 leader_election.cc:302] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:34.547515 4817 leader_election.cc:302] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:34.548187 4979 raft_consensus.cc:3037] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.548254 5104 raft_consensus.cc:2781] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.549134 5104 raft_consensus.cc:482] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.549135 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.549561 4908 raft_consensus.cc:2781] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.549976 4908 raft_consensus.cc:482] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.549960 5104 raft_consensus.cc:3037] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.550516 4908 raft_consensus.cc:3037] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.552791 4776 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.578047 4817 leader_election.cc:302] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:34.476374 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f9ceac69f5c42ed858143d4d8f7cce1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.585866 4775 raft_consensus.cc:2379] T 9f9ceac69f5c42ed858143d4d8f7cce1 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:34.586763 4979 raft_consensus.cc:2726] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.593058 4895 tablet_bootstrap.cc:654] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.598433 5104 raft_consensus.cc:504] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.603031 5104 leader_election.cc:288] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.606240 4592 catalog_manager.cc:5484] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:34.611797 4895 tablet_bootstrap.cc:492] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:34.612416 4895 ts_tablet_manager.cc:1367] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.110s user 0.020s sys 0.007s
I20240502 21:39:34.612421 4817 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.614045 4951 raft_consensus.cc:2781] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:39:34.614503 4951 raft_consensus.cc:482] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.614928 4951 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:34.615649 4895 raft_consensus.cc:348] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.616608 4895 raft_consensus.cc:374] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.616941 4895 raft_consensus.cc:729] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.618224 4895 consensus_queue.cc:260] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.624465 4951 raft_consensus.cc:504] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:34.630138 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:34.630841 4895 ts_tablet_manager.cc:1398] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.018s user 0.005s sys 0.007s
I20240502 21:39:34.631140 4817 leader_election.cc:302] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:34.633145 5108 raft_consensus.cc:3037] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.633651 4895 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:34.634596 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238a1f48ddb14387bc2aa4419ff8879e" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.635272 4775 raft_consensus.cc:3037] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.637106 4899 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:34.637630 4899 ts_tablet_manager.cc:1367] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.349s user 0.077s sys 0.109s
I20240502 21:39:34.640293 4899 raft_consensus.cc:348] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.640957 5108 raft_consensus.cc:2726] T 9f9ceac69f5c42ed858143d4d8f7cce1 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:34.646509 4899 raft_consensus.cc:374] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.646886 4899 raft_consensus.cc:729] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.648687 4899 consensus_queue.cc:260] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.652154 4899 ts_tablet_manager.cc:1398] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20240502 21:39:34.653661 4850 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.653864 4899 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:34.658907 4951 leader_election.cc:288] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:34.659591 4664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 51733.8us
I20240502 21:39:34.610968 4997 raft_consensus.cc:504] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.663466 4775 raft_consensus.cc:2454] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.672463 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.673409 4699 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.682394 4903 raft_consensus.cc:482] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.683076 4903 raft_consensus.cc:504] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.686115 4903 leader_election.cc:288] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.686520 5104 raft_consensus.cc:2726] T f05bc1efec5f4b95a68f7b46fdd719ad P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:39:34.680529 4908 raft_consensus.cc:504] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.697273 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:34.701709 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238a1f48ddb14387bc2aa4419ff8879e" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:34.704702 4737 leader_election.cc:341] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.698511 4908 leader_election.cc:288] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.705749 4903 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.707864 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:34.708602 4896 tablet_bootstrap.cc:492] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:34.709086 4896 ts_tablet_manager.cc:1367] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.433s user 0.125s sys 0.159s
I20240502 21:39:34.711190 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:34.711791 4697 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:34.716159 4896 raft_consensus.cc:348] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:34.716887 4737 leader_election.cc:341] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.716933 4896 raft_consensus.cc:374] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.719187 4896 raft_consensus.cc:729] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.722211 4697 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 2.
I20240502 21:39:34.719852 4896 consensus_queue.cc:260] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.702515 4997 leader_election.cc:288] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:34.725872 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a57b1b34a74513b44d19508bf21459" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.726521 4775 raft_consensus.cc:3037] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:34.728673 4908 raft_consensus.cc:482] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.729138 4908 raft_consensus.cc:504] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.731855 4896 ts_tablet_manager.cc:1398] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240502 21:39:34.732795 4896 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:34.739665 4979 consensus_queue.cc:1035] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.739930 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:34.740639 4696 raft_consensus.cc:3037] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.741809 5104 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.742293 5104 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.742822 4908 leader_election.cc:288] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.749938 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.744127 4817 leader_election.cc:302] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.753458 4951 raft_consensus.cc:2781] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.756088 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.756845 4773 raft_consensus.cc:2454] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:34.757906 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:34.764886 5104 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:34.752630 4818 leader_election.cc:341] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.767302 4818 leader_election.cc:302] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.751042 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79212407f8774e83a441fc94e7244824" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.768446 5104 raft_consensus.cc:2781] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:34.768829 4776 raft_consensus.cc:3037] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.768978 5104 raft_consensus.cc:686] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [term 2 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:34.769073 5108 raft_consensus.cc:482] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:34.758755 4697 raft_consensus.cc:3037] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.769730 5108 raft_consensus.cc:504] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.774492 4899 tablet_bootstrap.cc:654] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
W20240502 21:39:34.779474 4740 leader_election.cc:341] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.779654 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.782672 4951 raft_consensus.cc:686] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:34.788144 4817 leader_election.cc:302] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.793691 5073 consensus_queue.cc:1035] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:34.794368 4951 consensus_queue.cc:237] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.783815 4774 raft_consensus.cc:2454] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.786244 5104 consensus_queue.cc:237] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.786463 5108 leader_election.cc:288] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:34.780818 4740 leader_election.cc:302] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.784202 4997 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240502 21:39:34.815038 4775 raft_consensus.cc:2454] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 2.
I20240502 21:39:34.818315 4903 raft_consensus.cc:2726] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:39:34.820050 4664 consensus_peers.cc:485] T 263af6fed17b44af8fc338d6aa060bff P 09ff1ee782864ac99d05a7f011aeef44 -> Peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Couldn't send request to peer ddea410051614f23b93c7342ec7cc545. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:34.828568 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:34.821743 5108 raft_consensus.cc:2781] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:34.829249 5108 raft_consensus.cc:482] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:34.830091 5108 raft_consensus.cc:3037] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:34.809978 4696 raft_consensus.cc:2454] T c82b5ab84d5c47fd97758728b0244499 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:34.838868 4697 raft_consensus.cc:2454] T 79212407f8774e83a441fc94e7244824 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.839605 4776 raft_consensus.cc:2454] T 79212407f8774e83a441fc94e7244824 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:34.829476 4775 raft_consensus.cc:2379] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:34.843969 4979 consensus_queue.cc:1035] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 21:39:34.840063 4817 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.862687 5112 raft_consensus.cc:2781] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:39:34.863705 5112 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09ff1ee782864ac99d05a7f011aeef44)
I20240502 21:39:34.867581 5112 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:34.868543 4896 tablet_bootstrap.cc:654] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.876217 5108 raft_consensus.cc:504] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.878834 4817 leader_election.cc:302] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:34.879705 4895 tablet_bootstrap.cc:654] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:34.882107 5104 raft_consensus.cc:2781] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.882510 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.883734 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.883913 5104 raft_consensus.cc:686] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:34.884815 5104 consensus_queue.cc:237] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.885231 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302a987ffb0842c0bfae54675c665763" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:34.895278 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:34.903952 4593 catalog_manager.cc:5484] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:34.943049 4899 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:34.943668 4899 ts_tablet_manager.cc:1367] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.290s user 0.016s sys 0.007s
I20240502 21:39:34.903868 5108 leader_election.cc:288] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:34.952116 5116 log.cc:924] Time spent T 263af6fed17b44af8fc338d6aa060bff P df8e5a877b6640d692917083c9fb5282: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20240502 21:39:34.953344 4899 raft_consensus.cc:348] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.954144 4899 raft_consensus.cc:374] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:34.954449 4899 raft_consensus.cc:729] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:34.954506 4740 leader_election.cc:302] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:34.955148 4899 consensus_queue.cc:260] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:34.953826 4818 leader_election.cc:341] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.955788 4993 raft_consensus.cc:2781] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:34.957732 4993 raft_consensus.cc:686] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:34.958560 4993 consensus_queue.cc:237] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:34.960140 4818 leader_election.cc:302] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:34.966574 4899 ts_tablet_manager.cc:1398] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20240502 21:39:34.969472 4899 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:34.966349 5108 raft_consensus.cc:3037] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:34.990708 4818 leader_election.cc:341] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:34.997224 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:34.997941 4776 raft_consensus.cc:3037] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.009363 4593 catalog_manager.cc:5484] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:35.018770 4979 consensus_queue.cc:1035] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.025557 4896 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.026144 4896 ts_tablet_manager.cc:1367] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.294s user 0.018s sys 0.005s
I20240502 21:39:35.017488 5108 raft_consensus.cc:2726] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.041818 4896 raft_consensus.cc:348] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.042945 4896 raft_consensus.cc:374] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.043690 4896 raft_consensus.cc:729] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.047665 4895 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.034901 5112 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.024827 5071 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240502 21:39:35.048378 4895 ts_tablet_manager.cc:1367] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.415s user 0.016s sys 0.017s
I20240502 21:39:35.056566 4896 consensus_queue.cc:260] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.066753 5112 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.069231 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.071252 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.073417 4817 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:35.081300 4899 tablet_bootstrap.cc:654] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.053701 4979 consensus_queue.cc:1035] T 9f9ceac69f5c42ed858143d4d8f7cce1 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20240502 21:39:35.089668 4776 raft_consensus.cc:2454] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:35.048838 4850 raft_consensus.cc:1226] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Rejecting Update request from peer 09ff1ee782864ac99d05a7f011aeef44 for earlier term 1. Current term is 2. Ops: []
I20240502 21:39:35.091482 4817 leader_election.cc:302] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:35.094250 5108 raft_consensus.cc:2781] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.094759 5108 raft_consensus.cc:686] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:35.095603 5108 consensus_queue.cc:237] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.102074 5112 raft_consensus.cc:2726] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240502 21:39:35.106850 4662 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 52985.3us
I20240502 21:39:35.118616 5128 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.132620 4899 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.133160 4899 ts_tablet_manager.cc:1367] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.164s user 0.012s sys 0.006s
I20240502 21:39:35.139772 4899 raft_consensus.cc:348] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.139950 4979 consensus_queue.cc:1046] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.140569 4899 raft_consensus.cc:374] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.140872 4899 raft_consensus.cc:729] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.139848 4592 catalog_manager.cc:5484] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.141490 4899 consensus_queue.cc:260] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.142326 4895 raft_consensus.cc:348] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.142686 4592 catalog_manager.cc:5484] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 2, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 2 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.143553 4895 raft_consensus.cc:374] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.143366 4979 raft_consensus.cc:3032] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:39:35.143997 4895 raft_consensus.cc:729] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.144428 4979 raft_consensus.cc:729] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:35.144891 4895 consensus_queue.cc:260] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.145463 4979 consensus_queue.cc:260] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.147989 4979 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:35.149298 4899 ts_tablet_manager.cc:1398] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240502 21:39:35.159471 4899 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:35.150633 4895 ts_tablet_manager.cc:1398] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.084s user 0.003s sys 0.003s
I20240502 21:39:35.161089 4895 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.158133 4896 ts_tablet_manager.cc:1398] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.126s user 0.023s sys 0.023s
I20240502 21:39:35.165038 4896 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.170931 4896 tablet_bootstrap.cc:654] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.172508 4899 tablet_bootstrap.cc:654] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.172789 4895 tablet_bootstrap.cc:654] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.195291 5111 raft_consensus.cc:482] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.195892 5111 raft_consensus.cc:504] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.197932 5111 leader_election.cc:288] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.199805 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:35.201678 4740 leader_election.cc:341] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.202522 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:35.203950 4737 leader_election.cc:341] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.204389 4737 leader_election.cc:302] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.206315 5111 raft_consensus.cc:2726] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.208719 4594 catalog_manager.cc:5484] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.250197 4696 raft_consensus.cc:1261] T 07a57b1b34a74513b44d19508bf21459 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 21:39:35.252362 5104 consensus_queue.cc:1035] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.265796 4895 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.266630 4895 ts_tablet_manager.cc:1367] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.106s user 0.020s sys 0.020s
I20240502 21:39:35.265796 4899 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.268463 4899 ts_tablet_manager.cc:1367] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.109s user 0.029s sys 0.025s
I20240502 21:39:35.269789 4895 raft_consensus.cc:348] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.270587 4895 raft_consensus.cc:374] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.270931 4895 raft_consensus.cc:729] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.270778 4899 raft_consensus.cc:348] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.271492 4899 raft_consensus.cc:374] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.271770 4899 raft_consensus.cc:729] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.271771 4895 consensus_queue.cc:260] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.272542 4899 consensus_queue.cc:260] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.274238 4895 ts_tablet_manager.cc:1398] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:39:35.275321 4895 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.278650 4896 tablet_bootstrap.cc:492] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.279134 4896 ts_tablet_manager.cc:1367] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.114s user 0.033s sys 0.071s
I20240502 21:39:35.281520 4896 raft_consensus.cc:348] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.282187 4896 raft_consensus.cc:374] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.282449 4896 raft_consensus.cc:729] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.283015 4896 consensus_queue.cc:260] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.285413 4896 ts_tablet_manager.cc:1398] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:39:35.285311 4774 raft_consensus.cc:1261] T 07a57b1b34a74513b44d19508bf21459 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 21:39:35.286397 4896 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.287204 5104 consensus_queue.cc:1035] T 07a57b1b34a74513b44d19508bf21459 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.292629 4896 tablet_bootstrap.cc:654] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.296192 4899 ts_tablet_manager.cc:1398] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
I20240502 21:39:35.297488 4899 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:35.304104 4899 tablet_bootstrap.cc:654] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
W20240502 21:39:35.325886 4818 consensus_peers.cc:485] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:35.326918 4895 tablet_bootstrap.cc:654] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.337661 5110 consensus_queue.cc:1035] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.340828 4895 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.341584 4895 ts_tablet_manager.cc:1367] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.006s
I20240502 21:39:35.346822 4895 raft_consensus.cc:348] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.347921 4895 raft_consensus.cc:374] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.348369 4895 raft_consensus.cc:729] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.349406 4895 consensus_queue.cc:260] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.354331 5110 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.375030 4895 ts_tablet_manager.cc:1398] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.033s user 0.007s sys 0.001s
I20240502 21:39:35.405683 5112 consensus_queue.cc:1035] T 79212407f8774e83a441fc94e7244824 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.408131 4895 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.440922 4899 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.441460 4899 ts_tablet_manager.cc:1367] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.144s user 0.057s sys 0.049s
I20240502 21:39:35.441906 5104 raft_consensus.cc:482] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.442916 5104 raft_consensus.cc:504] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.444015 4899 raft_consensus.cc:348] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.456720 4899 raft_consensus.cc:374] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.457059 4899 raft_consensus.cc:729] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.456894 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.462532 5104 leader_election.cc:288] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.463433 4899 consensus_queue.cc:260] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:35.464093 4818 leader_election.cc:341] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.464527 4895 tablet_bootstrap.cc:654] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.465044 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.466065 4899 ts_tablet_manager.cc:1398] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20240502 21:39:35.467042 4899 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
W20240502 21:39:35.467628 4817 leader_election.cc:341] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.468603 4896 tablet_bootstrap.cc:492] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.468094 4817 leader_election.cc:302] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:35.469097 4896 ts_tablet_manager.cc:1367] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.183s user 0.077s sys 0.081s
I20240502 21:39:35.469324 5108 raft_consensus.cc:482] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.469880 5108 raft_consensus.cc:504] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.471505 4896 raft_consensus.cc:348] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.471968 5108 leader_election.cc:288] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.472234 4896 raft_consensus.cc:374] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.473264 4896 raft_consensus.cc:729] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.473347 5108 raft_consensus.cc:2726] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.473464 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab81ee4feb347d38912f2d2cee03fcd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.475217 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab81ee4feb347d38912f2d2cee03fcd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.475844 4771 raft_consensus.cc:2454] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:35.476902 4895 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.477435 4895 ts_tablet_manager.cc:1367] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.004s
I20240502 21:39:35.472851 4899 tablet_bootstrap.cc:654] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.477203 4817 leader_election.cc:302] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.478404 5104 raft_consensus.cc:2781] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.478837 5104 raft_consensus.cc:482] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.479195 5104 raft_consensus.cc:3037] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.473944 4896 consensus_queue.cc:260] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.481110 4696 raft_consensus.cc:2454] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:35.482758 4896 ts_tablet_manager.cc:1398] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240502 21:39:35.480600 4895 raft_consensus.cc:348] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.485421 4895 raft_consensus.cc:374] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.485785 4895 raft_consensus.cc:729] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.485715 5104 raft_consensus.cc:504] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.486662 4895 consensus_queue.cc:260] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.489745 4895 ts_tablet_manager.cc:1398] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 21:39:35.488050 4896 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.491837 5108 raft_consensus.cc:482] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.492358 5108 raft_consensus.cc:504] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.492916 4895 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.495355 5130 raft_consensus.cc:482] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.495853 5130 raft_consensus.cc:504] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.497004 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.497844 5130 leader_election.cc:288] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.497982 4993 consensus_queue.cc:1035] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.498670 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.499280 4771 raft_consensus.cc:2454] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:35.499828 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab81ee4feb347d38912f2d2cee03fcd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.500429 4696 raft_consensus.cc:3037] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.500957 4895 tablet_bootstrap.cc:654] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.494397 5108 leader_election.cc:288] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.503775 4817 leader_election.cc:302] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.505277 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.506105 4896 tablet_bootstrap.cc:654] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
W20240502 21:39:35.507481 4662 leader_election.cc:341] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.511075 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.511762 4771 raft_consensus.cc:2454] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
W20240502 21:39:35.512889 4818 leader_election.cc:341] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.514091 4951 raft_consensus.cc:2781] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.514568 4951 raft_consensus.cc:482] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.514969 4951 raft_consensus.cc:3037] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.515843 4696 raft_consensus.cc:2454] T 2ab81ee4feb347d38912f2d2cee03fcd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:35.516618 4664 leader_election.cc:302] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.519634 4979 raft_consensus.cc:2781] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.519994 4979 raft_consensus.cc:482] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.520346 4979 raft_consensus.cc:3037] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.526764 4951 raft_consensus.cc:504] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.535323 5129 raft_consensus.cc:482] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.536010 5129 raft_consensus.cc:504] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.537042 5104 leader_election.cc:288] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.538101 5129 leader_election.cc:288] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.538774 4951 leader_election.cc:288] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.541788 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.544245 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.544884 4774 raft_consensus.cc:3037] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.547111 4896 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.547677 4896 ts_tablet_manager.cc:1367] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.060s user 0.017s sys 0.009s
I20240502 21:39:35.550806 4895 tablet_bootstrap.cc:492] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.551276 4895 ts_tablet_manager.cc:1367] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.059s user 0.013s sys 0.011s
I20240502 21:39:35.551188 4896 raft_consensus.cc:348] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.551964 4896 raft_consensus.cc:374] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.552238 4896 raft_consensus.cc:729] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
W20240502 21:39:35.554066 4737 consensus_peers.cc:485] T c82b5ab84d5c47fd97758728b0244499 P ddea410051614f23b93c7342ec7cc545 -> Peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Couldn't send request to peer df8e5a877b6640d692917083c9fb5282. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:35.553926 4895 raft_consensus.cc:348] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.553978 4896 consensus_queue.cc:260] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.554915 4895 raft_consensus.cc:374] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.555233 4895 raft_consensus.cc:729] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.555869 4895 consensus_queue.cc:260] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.557837 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab81ee4feb347d38912f2d2cee03fcd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.558032 4895 ts_tablet_manager.cc:1398] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 21:39:35.558435 4771 raft_consensus.cc:3037] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.559190 4895 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.562703 4993 raft_consensus.cc:482] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.562747 4818 leader_election.cc:302] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.562629 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467ec4dc8d27420e9ea3fa4584153e65" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.563493 4993 raft_consensus.cc:504] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.564577 4773 raft_consensus.cc:2379] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:35.563894 5104 raft_consensus.cc:2781] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.565480 5104 raft_consensus.cc:686] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:35.566313 5104 consensus_queue.cc:237] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.567682 4896 ts_tablet_manager.cc:1398] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 21:39:35.570343 5108 raft_consensus.cc:482] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.570834 5108 raft_consensus.cc:504] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.572760 5108 leader_election.cc:288] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.575968 4896 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.581765 4993 leader_election.cc:288] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.585465 4895 tablet_bootstrap.cc:654] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.555492 4774 raft_consensus.cc:2454] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:35.583279 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.583879 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.584736 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.587306 4817 leader_election.cc:302] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.588032 4951 raft_consensus.cc:2781] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.588526 4951 raft_consensus.cc:686] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
W20240502 21:39:35.589046 4737 leader_election.cc:341] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.589418 4951 consensus_queue.cc:237] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.591307 4895 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.591852 4895 ts_tablet_manager.cc:1367] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
I20240502 21:39:35.587599 4776 raft_consensus.cc:2454] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:35.585808 5108 consensus_queue.cc:1035] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.594702 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.594949 4895 raft_consensus.cc:348] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.582173 4896 tablet_bootstrap.cc:654] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.595906 4895 raft_consensus.cc:374] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.596254 4895 raft_consensus.cc:729] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.597064 4895 consensus_queue.cc:260] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.599629 4899 tablet_bootstrap.cc:492] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.599767 4895 ts_tablet_manager.cc:1398] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 21:39:35.597303 4979 raft_consensus.cc:504] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:35.601177 4818 leader_election.cc:341] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.601339 4979 leader_election.cc:288] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.602144 4696 raft_consensus.cc:2379] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:35.602703 4817 leader_election.cc:302] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.603683 5104 raft_consensus.cc:2781] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.604054 5104 raft_consensus.cc:482] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.604414 5104 raft_consensus.cc:3037] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.604673 4895 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:35.607496 4664 leader_election.cc:302] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.600148 4899 ts_tablet_manager.cc:1367] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.011s
I20240502 21:39:35.608486 4979 raft_consensus.cc:3037] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.618742 4771 raft_consensus.cc:2454] T 2ab81ee4feb347d38912f2d2cee03fcd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:35.618834 4899 raft_consensus.cc:348] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.619997 4899 raft_consensus.cc:374] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.620298 4899 raft_consensus.cc:729] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.619818 4592 catalog_manager.cc:5484] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.620560 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.622630 4697 raft_consensus.cc:2454] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:35.621392 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:35.621017 4899 consensus_queue.cc:260] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.632018 4899 ts_tablet_manager.cc:1398] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20240502 21:39:35.632934 4899 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:35.638731 4979 raft_consensus.cc:2726] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.643388 4896 tablet_bootstrap.cc:492] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.643903 4896 ts_tablet_manager.cc:1367] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.005s
I20240502 21:39:35.646453 4896 raft_consensus.cc:348] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.647142 4896 raft_consensus.cc:374] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.647440 4896 raft_consensus.cc:729] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.647253 4740 leader_election.cc:302] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.649300 4993 raft_consensus.cc:2781] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.649677 4896 consensus_queue.cc:260] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.649677 4993 raft_consensus.cc:482] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.651072 4993 raft_consensus.cc:3037] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.663182 4896 ts_tablet_manager.cc:1398] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.019s user 0.007s sys 0.002s
I20240502 21:39:35.664006 4896 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.662808 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.692663 4896 tablet_bootstrap.cc:654] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.694263 4771 raft_consensus.cc:3037] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.705621 4951 consensus_queue.cc:1035] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:35.716845 4993 raft_consensus.cc:504] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.718828 5104 raft_consensus.cc:504] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.724901 4593 catalog_manager.cc:5484] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.727355 4993 leader_election.cc:288] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.727592 4850 raft_consensus.cc:2379] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:35.730266 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:35.730887 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:35.731779 4737 leader_election.cc:341] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.732594 4895 tablet_bootstrap.cc:654] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.735895 4899 tablet_bootstrap.cc:654] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.741659 5104 leader_election.cc:288] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.744081 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.744446 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e71664f51b64436906f9c21a6b46d89" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.744994 4699 raft_consensus.cc:2379] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:35.746376 4818 leader_election.cc:302] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:35.747879 4951 raft_consensus.cc:2726] T 9e71664f51b64436906f9c21a6b46d89 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.749401 4896 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.749570 4895 tablet_bootstrap.cc:492] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.749893 4896 ts_tablet_manager.cc:1367] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.002s
I20240502 21:39:35.750042 4895 ts_tablet_manager.cc:1367] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.146s user 0.007s sys 0.004s
I20240502 21:39:35.752434 4896 raft_consensus.cc:348] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.753162 4896 raft_consensus.cc:374] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.753456 4896 raft_consensus.cc:729] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.753329 4895 raft_consensus.cc:348] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.754127 4895 raft_consensus.cc:374] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.754441 4895 raft_consensus.cc:729] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.754259 4896 consensus_queue.cc:260] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.755123 4895 consensus_queue.cc:260] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.756600 5111 raft_consensus.cc:482] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.757010 4896 ts_tablet_manager.cc:1398] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240502 21:39:35.757115 5111 raft_consensus.cc:504] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.758005 4896 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.731510 4696 raft_consensus.cc:3037] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.759179 5111 leader_election.cc:288] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.760773 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.760875 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:35.763031 4740 leader_election.cc:341] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.763968 4896 tablet_bootstrap.cc:654] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.775486 4895 ts_tablet_manager.cc:1398] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20240502 21:39:35.776723 4895 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
W20240502 21:39:35.776993 4737 leader_election.cc:341] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.777448 4737 leader_election.cc:302] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.778198 5111 raft_consensus.cc:2726] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.764550 4771 raft_consensus.cc:2454] T 9e71664f51b64436906f9c21a6b46d89 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:35.783311 4664 leader_election.cc:302] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.784744 5130 raft_consensus.cc:2781] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.785235 5130 raft_consensus.cc:686] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:35.786013 4899 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.786502 4899 ts_tablet_manager.cc:1367] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.004s
I20240502 21:39:35.786028 5130 consensus_queue.cc:237] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.790719 4696 raft_consensus.cc:2454] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:35.796161 4740 leader_election.cc:302] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.797099 5111 raft_consensus.cc:2781] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.797596 5111 raft_consensus.cc:686] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:35.798470 5111 consensus_queue.cc:237] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.799749 4899 raft_consensus.cc:348] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.800509 4899 raft_consensus.cc:374] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.800772 4899 raft_consensus.cc:729] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.801362 4899 consensus_queue.cc:260] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.815415 4896 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:35.816095 4896 ts_tablet_manager.cc:1367] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.008s
I20240502 21:39:35.819528 4895 tablet_bootstrap.cc:654] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.824946 5111 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.825538 5111 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.827466 5111 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.820654 4896 raft_consensus.cc:348] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.827165 4592 catalog_manager.cc:5484] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:35.828182 4896 raft_consensus.cc:374] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.828505 5108 raft_consensus.cc:482] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.829130 5108 raft_consensus.cc:504] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.829440 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.830073 4850 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:35.831189 5108 leader_election.cc:288] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.828565 4896 raft_consensus.cc:729] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.832486 4896 consensus_queue.cc:260] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.833729 4737 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.834880 4896 ts_tablet_manager.cc:1398] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20240502 21:39:35.837617 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.837869 5111 raft_consensus.cc:2781] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:39:35.838429 5111 raft_consensus.cc:482] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.838361 4771 raft_consensus.cc:2454] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:35.840333 4979 raft_consensus.cc:482] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.840453 4899 ts_tablet_manager.cc:1398] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20240502 21:39:35.840809 4979 raft_consensus.cc:504] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.841243 4896 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:35.841341 4899 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:35.842823 4979 leader_election.cc:288] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.844049 4593 catalog_manager.cc:5484] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:35.845428 4817 leader_election.cc:302] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:35.844640 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.848853 4697 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:35.838948 5111 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:35.854126 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.854769 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.852829 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcdd2a6ae1204bfdb7814087ba453fed" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.850512 4951 raft_consensus.cc:2781] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.858942 4951 raft_consensus.cc:482] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.859314 4951 raft_consensus.cc:3037] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.863006 5111 raft_consensus.cc:504] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:35.864208 4818 leader_election.cc:341] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.864528 4771 raft_consensus.cc:2379] T bcdd2a6ae1204bfdb7814087ba453fed P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:35.865629 4664 leader_election.cc:302] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.866727 5125 raft_consensus.cc:3037] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.876669 4979 raft_consensus.cc:482] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.877169 4979 raft_consensus.cc:504] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.878968 4979 leader_election.cc:288] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.880774 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:35.882606 4850 raft_consensus.cc:2454] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:35.883172 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.883824 4662 leader_election.cc:302] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
W20240502 21:39:35.884932 4664 leader_election.cc:341] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.885708 4979 raft_consensus.cc:2781] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.886068 4979 raft_consensus.cc:482] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.886502 4979 raft_consensus.cc:3037] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.890383 4899 tablet_bootstrap.cc:654] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.890545 4896 tablet_bootstrap.cc:654] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.890293 4951 raft_consensus.cc:504] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.893038 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.893545 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06992601e8624b078b7f76f1d6495d96" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:35.893949 4696 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:35.894107 4850 raft_consensus.cc:3037] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:39:35.895095 4979 raft_consensus.cc:504] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.897765 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.898445 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.898110 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:35.899055 4774 raft_consensus.cc:3037] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.899755 4993 raft_consensus.cc:482] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.900225 4993 raft_consensus.cc:504] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.901481 4696 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 2.
I20240502 21:39:35.902176 4993 leader_election.cc:288] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.902992 5125 raft_consensus.cc:2726] T bcdd2a6ae1204bfdb7814087ba453fed P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.903359 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:35.898994 4818 leader_election.cc:341] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.904114 4740 leader_election.cc:302] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:35.904978 4993 raft_consensus.cc:2781] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:39:35.906165 4993 raft_consensus.cc:686] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [term 2 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:35.907117 4993 consensus_queue.cc:237] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:35.907634 4737 leader_election.cc:341] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.907974 4774 raft_consensus.cc:2454] T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:35.910872 4817 leader_election.cc:302] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:35.908401 4951 leader_election.cc:288] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:35.911689 5108 raft_consensus.cc:2781] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.909391 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:35.912379 5108 raft_consensus.cc:686] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:35.912535 4696 raft_consensus.cc:2454] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:35.913574 5108 consensus_queue.cc:237] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.919649 4850 raft_consensus.cc:2454] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 2.
I20240502 21:39:35.920234 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:35.921270 4979 leader_election.cc:288] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.921406 4849 raft_consensus.cc:3037] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.934034 4895 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:35.934566 4895 ts_tablet_manager.cc:1367] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.006s
I20240502 21:39:35.937372 4895 raft_consensus.cc:348] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.938278 4895 raft_consensus.cc:374] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.938601 4895 raft_consensus.cc:729] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.938654 4593 catalog_manager.cc:5484] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 2, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 2 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:35.939416 4592 catalog_manager.cc:5484] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:35.941890 4895 consensus_queue.cc:260] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.944598 4895 ts_tablet_manager.cc:1398] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20240502 21:39:35.945678 4895 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
W20240502 21:39:35.946969 4664 leader_election.cc:341] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.947932 4740 leader_election.cc:302] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:35.948738 4997 raft_consensus.cc:2781] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:35.949096 4997 raft_consensus.cc:482] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:35.949429 4997 raft_consensus.cc:3037] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:35.949493 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:35.950477 5108 raft_consensus.cc:482] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:35.951102 5108 raft_consensus.cc:504] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.954489 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:35.955693 4818 leader_election.cc:341] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.956574 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:35.948067 5111 leader_election.cc:288] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.956801 4997 raft_consensus.cc:504] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.957913 5108 leader_election.cc:288] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:35.958168 4817 leader_election.cc:341] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.958613 4817 leader_election.cc:302] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:35.956938 4849 raft_consensus.cc:2454] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:35.959183 4895 tablet_bootstrap.cc:654] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.960088 4662 leader_election.cc:302] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:35.960886 5125 raft_consensus.cc:2781] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:35.961333 5125 raft_consensus.cc:686] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:35.961632 4899 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:35.962294 4899 ts_tablet_manager.cc:1367] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.121s user 0.011s sys 0.000s
I20240502 21:39:35.962419 5125 consensus_queue.cc:237] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.966158 4899 raft_consensus.cc:348] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.967037 4899 raft_consensus.cc:374] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:35.967365 4899 raft_consensus.cc:729] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:35.967528 5108 raft_consensus.cc:2726] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:35.968246 4899 consensus_queue.cc:260] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:35.977651 4899 ts_tablet_manager.cc:1398] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20240502 21:39:35.978925 4899 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:35.981212 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:35.982189 4696 raft_consensus.cc:3037] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:35.984434 4997 leader_election.cc:288] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:35.990235 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
W20240502 21:39:35.992090 4737 leader_election.cc:341] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:35.997043 4951 consensus_queue.cc:1035] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:35.999228 4899 tablet_bootstrap.cc:654] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:35.999255 4951 consensus_queue.cc:1035] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 21:39:36.019330 4895 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:36.019835 4895 ts_tablet_manager.cc:1367] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20240502 21:39:36.020954 4696 raft_consensus.cc:2454] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:36.023345 4895 raft_consensus.cc:348] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.024149 4895 raft_consensus.cc:374] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.024456 4895 raft_consensus.cc:729] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.027268 4740 leader_election.cc:302] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.026583 4592 catalog_manager.cc:5484] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:36.030457 5111 raft_consensus.cc:2781] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:36.030992 5111 raft_consensus.cc:686] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:36.031847 5111 consensus_queue.cc:237] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.026651 4895 consensus_queue.cc:260] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.051990 4896 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:36.052549 4896 ts_tablet_manager.cc:1367] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.217s user 0.035s sys 0.023s
I20240502 21:39:36.052721 4899 tablet_bootstrap.cc:492] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:36.052667 4771 raft_consensus.cc:3037] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.053326 4899 ts_tablet_manager.cc:1367] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.016s
I20240502 21:39:36.062747 5104 raft_consensus.cc:482] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.063318 5104 raft_consensus.cc:504] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.065377 4899 raft_consensus.cc:348] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.068557 4899 raft_consensus.cc:374] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.069000 4899 raft_consensus.cc:729] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.069823 4899 consensus_queue.cc:260] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.077507 5104 leader_election.cc:288] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:36.079207 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.079914 4697 raft_consensus.cc:2379] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:36.081473 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6171c5f680040b9a3efc1f98c456ea9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:36.083722 4817 leader_election.cc:302] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:36.084924 5108 raft_consensus.cc:3037] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.089054 4895 ts_tablet_manager.cc:1398] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20240502 21:39:36.055657 4896 raft_consensus.cc:348] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.090114 4895 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:36.090303 4896 raft_consensus.cc:374] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.090596 4896 raft_consensus.cc:729] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.091300 4896 consensus_queue.cc:260] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.093549 4899 ts_tablet_manager.cc:1398] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.040s user 0.006s sys 0.002s
I20240502 21:39:36.095486 4899 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:36.098626 4896 ts_tablet_manager.cc:1398] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20240502 21:39:36.099592 4896 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:36.103199 5112 consensus_queue.cc:1035] T 2ab81ee4feb347d38912f2d2cee03fcd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:36.110752 4592 catalog_manager.cc:5484] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:36.141872 4895 tablet_bootstrap.cc:654] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.146528 4899 tablet_bootstrap.cc:654] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.147547 4951 raft_consensus.cc:482] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.148015 4997 raft_consensus.cc:482] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09ff1ee782864ac99d05a7f011aeef44)
I20240502 21:39:36.148530 4997 raft_consensus.cc:504] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.149085 5108 raft_consensus.cc:2726] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.150561 5110 raft_consensus.cc:482] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.148442 4951 raft_consensus.cc:504] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.151003 5110 raft_consensus.cc:504] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.152343 4951 leader_election.cc:288] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:36.153106 4896 tablet_bootstrap.cc:654] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.153007 5110 leader_election.cc:288] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:36.164284 4979 raft_consensus.cc:482] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.164942 4979 raft_consensus.cc:504] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.166061 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.166713 4979 leader_election.cc:288] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.164047 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.169162 4696 raft_consensus.cc:2379] T 142da03edd3d4648aae37b78118070b5 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:36.169174 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:36.170922 4662 leader_election.cc:341] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:36.162550 5175 log.cc:924] Time spent T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Append to log took a long time: real 0.073s user 0.000s sys 0.004s
I20240502 21:39:36.172503 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142da03edd3d4648aae37b78118070b5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:36.174747 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:36.175890 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:36.177510 4817 leader_election.cc:341] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:36.179172 4664 leader_election.cc:341] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.179504 4664 leader_election.cc:302] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.180660 5130 raft_consensus.cc:2726] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.180817 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:36.182665 4818 leader_election.cc:341] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.184098 4818 leader_election.cc:302] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:36.185565 4997 leader_election.cc:288] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.186776 5110 raft_consensus.cc:2726] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.205857 4896 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:36.206485 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "263af6fed17b44af8fc338d6aa060bff" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.208882 4737 leader_election.cc:302] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.210638 5177 raft_consensus.cc:2726] T 263af6fed17b44af8fc338d6aa060bff P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:39:36.208109 5178 consensus_queue.cc:1035] T bcdd2a6ae1204bfdb7814087ba453fed P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240502 21:39:36.218397 4899 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:36.218989 4899 ts_tablet_manager.cc:1367] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.011s
I20240502 21:39:36.219146 5177 raft_consensus.cc:482] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.219905 5177 raft_consensus.cc:504] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.221522 4899 raft_consensus.cc:348] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.222225 4899 raft_consensus.cc:374] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.222501 4899 raft_consensus.cc:729] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.223134 4899 consensus_queue.cc:260] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.207691 4896 ts_tablet_manager.cc:1367] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.004s
I20240502 21:39:36.224570 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.225638 4850 raft_consensus.cc:2379] T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:36.226682 4896 raft_consensus.cc:348] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.227384 4896 raft_consensus.cc:374] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.227447 5177 leader_election.cc:288] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.228705 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7702c7f8653d498685b0f0e158594b3d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.230051 4817 leader_election.cc:302] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:36.231025 5178 raft_consensus.cc:3037] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.231024 4740 leader_election.cc:302] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.234802 4896 raft_consensus.cc:729] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.234978 5111 raft_consensus.cc:3037] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.235757 4896 consensus_queue.cc:260] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.240160 4899 ts_tablet_manager.cc:1398] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20240502 21:39:36.241447 4899 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:36.264667 5178 raft_consensus.cc:2726] T 142da03edd3d4648aae37b78118070b5 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.270107 5130 consensus_queue.cc:1035] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:36.271814 5111 raft_consensus.cc:2726] T 7702c7f8653d498685b0f0e158594b3d P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.308462 4896 ts_tablet_manager.cc:1398] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.084s user 0.008s sys 0.000s
I20240502 21:39:36.309952 4896 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:36.315199 4694 raft_consensus.cc:3037] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.326357 5111 consensus_queue.cc:1035] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:36.335765 4895 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:36.336462 4895 ts_tablet_manager.cc:1367] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.247s user 0.057s sys 0.063s
I20240502 21:39:36.336726 4979 consensus_queue.cc:1035] T 9e71664f51b64436906f9c21a6b46d89 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:36.337785 4899 tablet_bootstrap.cc:654] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.338922 4895 raft_consensus.cc:348] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.339700 4895 raft_consensus.cc:374] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.339993 4895 raft_consensus.cc:729] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.341578 4895 consensus_queue.cc:260] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.343178 4895 ts_tablet_manager.cc:1398] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240502 21:39:36.344905 4895 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:36.359109 4896 tablet_bootstrap.cc:654] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.361663 5130 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.362396 5130 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:36.363826 4818 consensus_peers.cc:485] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:36.367218 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.367913 4849 raft_consensus.cc:2454] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 2.
I20240502 21:39:36.369434 4662 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:36.370486 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:36.371270 4979 raft_consensus.cc:2781] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 21:39:36.371620 4979 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:36.371941 4979 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:36.377633 4979 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.365810 5130 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.372000 4895 tablet_bootstrap.cc:654] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.381649 4979 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 3 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.380610 5111 raft_consensus.cc:482] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.385023 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:36.385165 5111 raft_consensus.cc:504] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.385939 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:36.386723 4849 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Advancing to term 3
I20240502 21:39:36.393637 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.395262 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:36.397645 4740 leader_election.cc:341] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:36.399116 4737 leader_election.cc:341] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.399559 4737 leader_election.cc:302] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.401125 5177 raft_consensus.cc:2726] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.391281 5111 leader_election.cc:288] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.417378 5104 consensus_queue.cc:1035] T 9166e2d9377e4adb8fdbb58e8641e2a1 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:36.408802 4993 consensus_queue.cc:1035] T 142da03edd3d4648aae37b78118070b5 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240502 21:39:36.413188 4849 raft_consensus.cc:2454] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 3.
I20240502 21:39:36.425338 4662 leader_election.cc:302] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:36.433395 4979 raft_consensus.cc:2781] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 21:39:36.434067 4979 raft_consensus.cc:686] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [term 3 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:36.435029 4979 consensus_queue.cc:237] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.468336 4899 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:36.474640 4899 ts_tablet_manager.cc:1367] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.228s user 0.020s sys 0.000s
I20240502 21:39:36.478574 4896 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:36.479113 4896 ts_tablet_manager.cc:1367] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.170s user 0.017s sys 0.005s
I20240502 21:39:36.483119 4899 raft_consensus.cc:348] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.483938 4899 raft_consensus.cc:374] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.484388 4899 raft_consensus.cc:729] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.485070 4899 consensus_queue.cc:260] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.487200 4899 ts_tablet_manager.cc:1398] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240502 21:39:36.488083 4899 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:36.488749 4896 raft_consensus.cc:348] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.489425 4896 raft_consensus.cc:374] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.489708 4896 raft_consensus.cc:729] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.492576 4993 raft_consensus.cc:482] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.493199 4993 raft_consensus.cc:504] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.495079 4697 raft_consensus.cc:1261] T 06992601e8624b078b7f76f1d6495d96 P 09ff1ee782864ac99d05a7f011aeef44 [term 2 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 21:39:36.495658 4993 leader_election.cc:288] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.529167 4997 raft_consensus.cc:482] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.596280 4993 raft_consensus.cc:482] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09ff1ee782864ac99d05a7f011aeef44)
I20240502 21:39:36.597029 4993 raft_consensus.cc:504] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.502147 5190 consensus_queue.cc:1035] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.096s
I20240502 21:39:36.499255 4896 consensus_queue.cc:260] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.533865 5129 consensus_queue.cc:1035] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.066s
I20240502 21:39:36.600664 4896 ts_tablet_manager.cc:1398] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.121s user 0.008s sys 0.004s
I20240502 21:39:36.496598 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.601599 4896 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:36.602363 4993 leader_election.cc:288] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.503120 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.604382 4849 raft_consensus.cc:2454] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:36.596948 4997 raft_consensus.cc:504] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.607228 4592 catalog_manager.cc:5484] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:36.608124 4997 leader_election.cc:288] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.498747 4895 tablet_bootstrap.cc:492] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:36.610724 4895 ts_tablet_manager.cc:1367] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.266s user 0.024s sys 0.036s
I20240502 21:39:36.611152 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.613209 4895 raft_consensus.cc:348] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.613925 4895 raft_consensus.cc:374] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.614202 4895 raft_consensus.cc:729] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.594851 4899 tablet_bootstrap.cc:654] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.508765 5111 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.115s
I20240502 21:39:36.617295 4895 consensus_queue.cc:260] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.623247 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a3ab558007a45a7b1e07e09a6efc408" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.622225 4847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.673616 4895 ts_tablet_manager.cc:1398] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.063s user 0.003s sys 0.004s
I20240502 21:39:36.675298 4895 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:36.618733 4737 leader_election.cc:302] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
W20240502 21:39:36.697121 4740 leader_election.cc:341] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.623912 5177 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
W20240502 21:39:36.700503 4740 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 80174.8us
I20240502 21:39:36.698352 4849 raft_consensus.cc:2354] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 for earlier term 2. Current term is 3.
I20240502 21:39:36.696408 4851 raft_consensus.cc:1261] T 06992601e8624b078b7f76f1d6495d96 P df8e5a877b6640d692917083c9fb5282 [term 2 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 21:39:36.711345 4896 tablet_bootstrap.cc:654] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
W20240502 21:39:36.702176 4737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 83559.6us
I20240502 21:39:36.705404 5198 raft_consensus.cc:2781] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:36.720669 5198 raft_consensus.cc:482] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:36.721073 5198 raft_consensus.cc:3037] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.724110 4895 tablet_bootstrap.cc:654] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.737742 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.739046 4899 tablet_bootstrap.cc:492] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:36.739498 4899 ts_tablet_manager.cc:1367] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.252s user 0.036s sys 0.032s
I20240502 21:39:36.742161 5125 consensus_queue.cc:1035] T 7702c7f8653d498685b0f0e158594b3d P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:36.744774 4899 raft_consensus.cc:348] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.745471 4899 raft_consensus.cc:374] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.745738 4899 raft_consensus.cc:729] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
W20240502 21:39:36.746632 4737 leader_election.cc:341] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.746344 4899 consensus_queue.cc:260] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.747570 5190 consensus_queue.cc:1035] T 06992601e8624b078b7f76f1d6495d96 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:36.747865 4737 leader_election.cc:398] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259) with higher term. Message: Invalid argument: T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 for earlier term 2. Current term is 3.
I20240502 21:39:36.748901 4737 leader_election.cc:401] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240502 21:39:36.749118 4899 ts_tablet_manager.cc:1398] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240502 21:39:36.750169 4899 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:36.750641 5190 raft_consensus.cc:3037] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Advancing to term 3
I20240502 21:39:36.757783 5071 raft_consensus.cc:482] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.759155 5071 raft_consensus.cc:504] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.761385 5071 leader_election.cc:288] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:36.768553 4740 leader_election.cc:341] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.770318 4740 leader_election.cc:302] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:36.776746 5178 raft_consensus.cc:482] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.786680 4899 tablet_bootstrap.cc:654] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:36.779203 5198 raft_consensus.cc:504] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.799494 5111 raft_consensus.cc:2726] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:36.804244 4896 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:36.805326 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.806535 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:36.809192 4896 ts_tablet_manager.cc:1367] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.208s user 0.004s sys 0.016s
W20240502 21:39:36.814527 5200 log.cc:924] Time spent T 7702c7f8653d498685b0f0e158594b3d P df8e5a877b6640d692917083c9fb5282: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20240502 21:39:36.785504 5178 raft_consensus.cc:504] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.816397 5198 leader_election.cc:288] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.915448 4899 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
W20240502 21:39:36.907168 4883 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.3.113.254:39185, user_credentials={real_user=slave}} blocked reactor thread for 80518.5us
I20240502 21:39:36.901301 5190 raft_consensus.cc:2726] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259) with higher term. Message: Invalid argument: T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 for earlier term 2. Current term is 3.
I20240502 21:39:36.948951 4899 ts_tablet_manager.cc:1367] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.199s user 0.015s sys 0.014s
W20240502 21:39:36.916558 4817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 102465us
I20240502 21:39:36.904444 4993 raft_consensus.cc:482] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:36.950351 4993 raft_consensus.cc:504] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.951490 4899 raft_consensus.cc:348] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.952229 4899 raft_consensus.cc:374] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.842779 4850 raft_consensus.cc:2454] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:36.952579 4899 raft_consensus.cc:729] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.953308 4899 consensus_queue.cc:260] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.954416 5178 leader_election.cc:288] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:36.951428 4770 raft_consensus.cc:1261] T 1a3ab558007a45a7b1e07e09a6efc408 P ddea410051614f23b93c7342ec7cc545 [term 3 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 21:39:36.942581 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:36.955523 4899 ts_tablet_manager.cc:1398] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:39:36.955518 4849 raft_consensus.cc:3037] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:36.957746 4899 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:36.960783 4993 leader_election.cc:288] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:36.822580 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
W20240502 21:39:36.916013 4818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 92486.5us
I20240502 21:39:36.962256 4896 raft_consensus.cc:348] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.929015 4895 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:36.963032 4896 raft_consensus.cc:374] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:36.963366 4896 raft_consensus.cc:729] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:36.963445 4895 ts_tablet_manager.cc:1367] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.288s user 0.011s sys 0.010s
W20240502 21:39:36.893095 5196 log.cc:924] Time spent T 9166e2d9377e4adb8fdbb58e8641e2a1 P ddea410051614f23b93c7342ec7cc545: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20240502 21:39:36.971261 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:36.971992 4850 raft_consensus.cc:2454] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:36.972021 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:36.972623 4740 leader_election.cc:341] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.972985 5129 consensus_queue.cc:1035] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:36.973246 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:36.973893 4694 raft_consensus.cc:2454] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
W20240502 21:39:36.975704 4664 leader_election.cc:341] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:36.964115 4896 consensus_queue.cc:260] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.980610 4737 leader_election.cc:302] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:36.983078 4895 raft_consensus.cc:348] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.985435 4896 ts_tablet_manager.cc:1398] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.037s user 0.004s sys 0.002s
I20240502 21:39:37.001273 4895 raft_consensus.cc:374] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.001677 4895 raft_consensus.cc:729] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.002013 4896 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.002305 4895 consensus_queue.cc:260] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.999205 4662 leader_election.cc:302] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:37.004484 4895 ts_tablet_manager.cc:1398] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20240502 21:39:37.005095 5071 raft_consensus.cc:2781] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.005313 4895 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.005512 5071 raft_consensus.cc:482] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.005901 5071 raft_consensus.cc:3037] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.009510 4896 tablet_bootstrap.cc:654] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.009850 4899 tablet_bootstrap.cc:654] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.013486 5071 raft_consensus.cc:504] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:36.975227 4849 raft_consensus.cc:2454] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:37.017035 5190 raft_consensus.cc:2781] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.041425 5190 raft_consensus.cc:482] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.041927 5190 raft_consensus.cc:3037] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:37.042035 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:37.017679 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c82b5ab84d5c47fd97758728b0244499" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.018498 4895 tablet_bootstrap.cc:654] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.032526 4896 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.052161 4896 ts_tablet_manager.cc:1367] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.002s
I20240502 21:39:37.052048 5190 raft_consensus.cc:504] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.054894 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:37.055552 4695 raft_consensus.cc:3037] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.054894 4896 raft_consensus.cc:348] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.056790 4896 raft_consensus.cc:374] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.057073 4896 raft_consensus.cc:729] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.057704 4896 consensus_queue.cc:260] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.060075 4896 ts_tablet_manager.cc:1398] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:39:37.061094 4896 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.038869 4899 tablet_bootstrap.cc:492] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:37.061189 5071 leader_election.cc:288] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.061991 4851 raft_consensus.cc:1261] T 1a3ab558007a45a7b1e07e09a6efc408 P df8e5a877b6640d692917083c9fb5282 [term 3 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240502 21:39:37.066401 4695 raft_consensus.cc:2454] T 64a461c860474b8b9480a44b6f96c2d2 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:37.063668 4737 leader_election.cc:302] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
I20240502 21:39:37.069017 4899 ts_tablet_manager.cc:1367] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.010s
I20240502 21:39:37.070916 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:37.071743 4993 raft_consensus.cc:2781] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.072243 4993 raft_consensus.cc:686] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:37.076505 4740 leader_election.cc:302] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:37.077340 5111 raft_consensus.cc:2781] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.077863 5111 raft_consensus.cc:686] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:37.080777 4817 leader_election.cc:302] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:37.086194 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e05551863b345f78f90e88220cbb163" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:37.086475 4979 consensus_queue.cc:1035] T 1a3ab558007a45a7b1e07e09a6efc408 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:37.086954 4848 raft_consensus.cc:3037] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.088513 4993 consensus_queue.cc:237] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.086246 5190 leader_election.cc:288] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.091920 5111 consensus_queue.cc:237] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.096114 4899 raft_consensus.cc:348] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.097260 4899 raft_consensus.cc:374] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.097693 4899 raft_consensus.cc:729] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.100880 4899 consensus_queue.cc:260] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.090096 5112 raft_consensus.cc:2726] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.107664 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a461c860474b8b9480a44b6f96c2d2" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:37.108335 4849 raft_consensus.cc:3037] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.108536 4775 raft_consensus.cc:3037] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.192140 4850 raft_consensus.cc:3037] T c82b5ab84d5c47fd97758728b0244499 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.194787 4775 raft_consensus.cc:2454] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:37.198196 4896 tablet_bootstrap.cc:654] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.199270 4849 raft_consensus.cc:2454] T 64a461c860474b8b9480a44b6f96c2d2 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:37.201557 4848 raft_consensus.cc:2454] T 9e05551863b345f78f90e88220cbb163 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:37.204936 4662 leader_election.cc:302] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.206337 5125 raft_consensus.cc:2781] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.206174 4593 catalog_manager.cc:5484] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:37.209362 4895 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.209829 4895 ts_tablet_manager.cc:1367] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.205s user 0.007s sys 0.012s
I20240502 21:39:37.206908 5125 raft_consensus.cc:686] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:37.214123 5125 consensus_queue.cc:237] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.213726 4895 raft_consensus.cc:348] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.215731 4895 raft_consensus.cc:374] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.216037 4895 raft_consensus.cc:729] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.216722 4895 consensus_queue.cc:260] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.221520 5129 raft_consensus.cc:482] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.222034 5129 raft_consensus.cc:504] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.226900 4899 ts_tablet_manager.cc:1398] T 9e05551863b345f78f90e88220cbb163 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.158s user 0.009s sys 0.000s
I20240502 21:39:37.227860 4899 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:37.226941 5129 leader_election.cc:288] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.232154 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.233271 4895 ts_tablet_manager.cc:1398] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240502 21:39:37.232869 4770 raft_consensus.cc:2454] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:37.235605 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:37.237934 4662 leader_election.cc:341] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.250605 4895 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.251958 4664 leader_election.cc:302] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.255611 5071 raft_consensus.cc:2781] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.255988 5071 raft_consensus.cc:482] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.256332 5071 raft_consensus.cc:3037] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.279078 4899 tablet_bootstrap.cc:654] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.281489 4592 catalog_manager.cc:5484] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:37.291404 4896 tablet_bootstrap.cc:492] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.291956 4896 ts_tablet_manager.cc:1367] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.231s user 0.017s sys 0.018s
I20240502 21:39:37.295313 4896 raft_consensus.cc:348] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.296087 4896 raft_consensus.cc:374] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.296422 4896 raft_consensus.cc:729] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.297374 4896 consensus_queue.cc:260] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.299163 4896 ts_tablet_manager.cc:1398] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:39:37.305076 4896 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.321472 4895 tablet_bootstrap.cc:654] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.327209 4896 tablet_bootstrap.cc:654] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.329159 4993 raft_consensus.cc:482] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.329655 4993 raft_consensus.cc:504] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.331619 4993 leader_election.cc:288] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.333436 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:37.335846 4737 leader_election.cc:341] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.336815 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:37.338975 4740 leader_election.cc:341] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.339373 4740 leader_election.cc:302] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.344544 5190 raft_consensus.cc:2726] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.354692 5071 raft_consensus.cc:504] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.363772 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:37.364548 4775 raft_consensus.cc:3037] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.366340 5071 leader_election.cc:288] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.368067 4895 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.366369 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:37.368677 4895 ts_tablet_manager.cc:1367] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.118s user 0.017s sys 0.005s
I20240502 21:39:37.371969 4899 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:37.372486 4899 ts_tablet_manager.cc:1367] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.145s user 0.016s sys 0.001s
I20240502 21:39:37.373376 4895 raft_consensus.cc:348] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.374079 4895 raft_consensus.cc:374] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.374380 4895 raft_consensus.cc:729] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
W20240502 21:39:37.375367 4662 leader_election.cc:341] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.375069 4895 consensus_queue.cc:260] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.377772 4899 raft_consensus.cc:348] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.378489 4899 raft_consensus.cc:374] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.378793 4899 raft_consensus.cc:729] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.377882 4594 catalog_manager.cc:5484] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:37.379477 4899 consensus_queue.cc:260] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.381434 4896 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.381918 4896 ts_tablet_manager.cc:1367] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.007s
I20240502 21:39:37.382051 4899 ts_tablet_manager.cc:1398] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240502 21:39:37.382980 4899 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:37.384451 4896 raft_consensus.cc:348] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.385121 4896 raft_consensus.cc:374] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.385394 4896 raft_consensus.cc:729] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.385977 4896 consensus_queue.cc:260] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.388077 4896 ts_tablet_manager.cc:1398] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240502 21:39:37.388621 4899 tablet_bootstrap.cc:654] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.390168 4896 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.393518 4775 raft_consensus.cc:2454] T f0fda9381a074590aec89148d7b517cf P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:37.394263 4895 ts_tablet_manager.cc:1398] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20240502 21:39:37.394661 4664 leader_election.cc:302] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.396121 5071 raft_consensus.cc:2781] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.396639 5071 raft_consensus.cc:686] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:37.397464 5071 consensus_queue.cc:237] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.403095 4896 tablet_bootstrap.cc:654] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.404527 4895 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.411239 4895 tablet_bootstrap.cc:654] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.411182 4896 tablet_bootstrap.cc:492] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.412242 4896 ts_tablet_manager.cc:1367] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.006s
I20240502 21:39:37.413137 4899 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:37.413587 4899 ts_tablet_manager.cc:1367] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20240502 21:39:37.414739 4896 raft_consensus.cc:348] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.415546 4896 raft_consensus.cc:374] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.415823 4896 raft_consensus.cc:729] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.415900 4899 raft_consensus.cc:348] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.416661 4899 raft_consensus.cc:374] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.416486 4896 consensus_queue.cc:260] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.416930 4899 raft_consensus.cc:729] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.417647 4899 consensus_queue.cc:260] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.418470 5129 raft_consensus.cc:482] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.419028 5129 raft_consensus.cc:504] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.420101 4896 ts_tablet_manager.cc:1398] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:39:37.421036 4896 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.421430 5129 leader_election.cc:288] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.423099 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.423699 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd1bcf32e7444ed80e0f7ae05786dde" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:37.425179 4850 raft_consensus.cc:2379] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09ff1ee782864ac99d05a7f011aeef44 in current term 1: Already voted for candidate ddea410051614f23b93c7342ec7cc545 in this term.
I20240502 21:39:37.426671 4896 tablet_bootstrap.cc:654] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.426631 4662 leader_election.cc:302] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.429685 5125 raft_consensus.cc:3037] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.430995 4899 ts_tablet_manager.cc:1398] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240502 21:39:37.432557 4899 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:37.443110 4895 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.443714 4895 ts_tablet_manager.cc:1367] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.004s
I20240502 21:39:37.445653 5125 raft_consensus.cc:2726] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.446354 4895 raft_consensus.cc:348] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.446966 4895 raft_consensus.cc:374] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.447187 4895 raft_consensus.cc:729] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.447734 4895 consensus_queue.cc:260] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.455188 4951 raft_consensus.cc:482] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.455884 4951 raft_consensus.cc:504] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.461009 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.462024 4694 raft_consensus.cc:2454] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:37.464251 4818 leader_election.cc:302] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.465147 5104 raft_consensus.cc:2781] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.465497 5104 raft_consensus.cc:482] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.465821 5104 raft_consensus.cc:3037] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.475250 4951 leader_election.cc:288] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.475019 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.476409 4775 raft_consensus.cc:2454] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:37.525414 4594 catalog_manager.cc:5484] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:37.491894 5104 raft_consensus.cc:504] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.528874 4895 ts_tablet_manager.cc:1398] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.085s user 0.024s sys 0.024s
I20240502 21:39:37.529918 4895 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.533097 4899 tablet_bootstrap.cc:654] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.536831 5190 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:37.541289 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:37.541932 4694 raft_consensus.cc:3037] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.543288 5110 raft_consensus.cc:482] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.543964 5110 raft_consensus.cc:504] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.551724 4896 tablet_bootstrap.cc:492] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.552235 4896 ts_tablet_manager.cc:1367] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.131s user 0.029s sys 0.022s
I20240502 21:39:37.552713 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.553695 4695 raft_consensus.cc:2454] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:37.554724 4896 raft_consensus.cc:348] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.555440 4896 raft_consensus.cc:374] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.555511 4818 leader_election.cc:302] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.557116 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2fb6c68b641a0a4e1ab5356b23c3d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:37.557722 4775 raft_consensus.cc:3037] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.558372 5178 raft_consensus.cc:2781] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.558773 5178 raft_consensus.cc:482] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.559098 5178 raft_consensus.cc:3037] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.559911 4895 tablet_bootstrap.cc:654] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.559999 5071 raft_consensus.cc:482] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.560556 5071 raft_consensus.cc:504] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.562541 5071 leader_election.cc:288] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.562734 5104 leader_election.cc:288] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.564276 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:37.555928 4896 raft_consensus.cc:729] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.566045 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9166e2d9377e4adb8fdbb58e8641e2a1" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.566869 5110 leader_election.cc:288] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.572719 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.569085 5178 raft_consensus.cc:504] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.573808 4775 raft_consensus.cc:2454] T 67a2fb6c68b641a0a4e1ab5356b23c3d P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.573760 4770 raft_consensus.cc:2454] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:37.565763 4896 consensus_queue.cc:260] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.575976 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:37.576596 4896 ts_tablet_manager.cc:1398] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20240502 21:39:37.576661 4695 raft_consensus.cc:3037] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.577291 4662 leader_election.cc:302] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.578307 5129 raft_consensus.cc:2726] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.578058 4896 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.583284 4817 leader_election.cc:302] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.587085 4896 tablet_bootstrap.cc:654] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.588709 5104 raft_consensus.cc:2781] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.589435 5104 raft_consensus.cc:686] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:37.590581 5104 consensus_queue.cc:237] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.596343 5190 consensus_queue.cc:1035] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:37.597975 4993 consensus_queue.cc:1035] T 64a461c860474b8b9480a44b6f96c2d2 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:37.609216 5178 leader_election.cc:288] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.611232 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d93565463749aab6550e43871f7730" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:37.611930 4775 raft_consensus.cc:3037] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.618919 4775 raft_consensus.cc:2454] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.622156 4817 leader_election.cc:302] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.622593 4694 raft_consensus.cc:2454] T 67a2fb6c68b641a0a4e1ab5356b23c3d P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.623580 5110 raft_consensus.cc:2781] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.627694 5110 raft_consensus.cc:686] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:37.629504 5110 consensus_queue.cc:237] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.635716 4695 raft_consensus.cc:2454] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.659965 4896 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.660588 4896 ts_tablet_manager.cc:1367] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.017s
I20240502 21:39:37.663728 4896 raft_consensus.cc:348] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.667141 4899 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:37.667685 4899 ts_tablet_manager.cc:1367] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.235s user 0.012s sys 0.015s
I20240502 21:39:37.670243 4899 raft_consensus.cc:348] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.670938 4899 raft_consensus.cc:374] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.671214 4899 raft_consensus.cc:729] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.667167 4896 raft_consensus.cc:374] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.673544 4896 raft_consensus.cc:729] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.674161 4896 consensus_queue.cc:260] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.675634 4896 ts_tablet_manager.cc:1398] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.015s user 0.003s sys 0.001s
I20240502 21:39:37.677829 4895 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.678362 4895 ts_tablet_manager.cc:1367] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.149s user 0.040s sys 0.050s
I20240502 21:39:37.680908 4895 raft_consensus.cc:348] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.681623 4895 raft_consensus.cc:374] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.681905 4895 raft_consensus.cc:729] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.682512 4895 consensus_queue.cc:260] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.685122 4896 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.690384 4896 tablet_bootstrap.cc:654] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.692755 4895 ts_tablet_manager.cc:1398] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20240502 21:39:37.693711 4895 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.699316 4895 tablet_bootstrap.cc:654] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.671826 4899 consensus_queue.cc:260] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.702033 4899 ts_tablet_manager.cc:1398] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20240502 21:39:37.703585 4899 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:37.708843 4899 tablet_bootstrap.cc:654] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.716902 4895 tablet_bootstrap.cc:492] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.717376 4895 ts_tablet_manager.cc:1367] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.006s
I20240502 21:39:37.719730 4895 raft_consensus.cc:348] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.729688 4895 raft_consensus.cc:374] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.730044 4895 raft_consensus.cc:729] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.739130 4896 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.740114 4896 ts_tablet_manager.cc:1367] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.002s
I20240502 21:39:37.737139 4895 consensus_queue.cc:260] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.742784 4896 raft_consensus.cc:348] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.743466 4896 raft_consensus.cc:374] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.743803 4896 raft_consensus.cc:729] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.747717 5111 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:37.750736 4895 ts_tablet_manager.cc:1398] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.033s user 0.003s sys 0.004s
I20240502 21:39:37.744503 4896 consensus_queue.cc:260] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.754900 4896 ts_tablet_manager.cc:1398] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20240502 21:39:37.756297 4895 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.759382 5129 raft_consensus.cc:482] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.760015 5129 raft_consensus.cc:504] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.761066 5104 raft_consensus.cc:482] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.761127 5190 raft_consensus.cc:482] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.761631 5104 raft_consensus.cc:504] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.763538 5129 leader_election.cc:288] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.757768 4899 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:37.765540 4899 ts_tablet_manager.cc:1367] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.000s
I20240502 21:39:37.761709 5190 raft_consensus.cc:504] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.767712 5190 leader_election.cc:288] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.768251 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:37.757777 4896 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
W20240502 21:39:37.770284 4737 leader_election.cc:341] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.770356 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a911c9636f426e9d08bee71f60eaab" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:37.768333 5104 leader_election.cc:288] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.771826 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.772528 4774 raft_consensus.cc:2454] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
W20240502 21:39:37.773494 4662 leader_election.cc:341] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.773674 4817 leader_election.cc:302] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.774448 5110 raft_consensus.cc:2781] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.774886 5110 raft_consensus.cc:482] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.775308 5110 raft_consensus.cc:3037] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.774220 4593 catalog_manager.cc:5484] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:37.779615 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.781153 4699 raft_consensus.cc:2454] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:37.801911 4896 tablet_bootstrap.cc:654] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.804162 4895 tablet_bootstrap.cc:654] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.797240 4899 raft_consensus.cc:348] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.805869 4899 raft_consensus.cc:374] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.806118 4899 raft_consensus.cc:729] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.807384 4593 catalog_manager.cc:5484] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:37.781982 5071 consensus_queue.cc:1035] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20240502 21:39:37.806684 4899 consensus_queue.cc:260] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.781584 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.815980 4899 ts_tablet_manager.cc:1398] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20240502 21:39:37.817348 4899 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:37.782245 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a911c9636f426e9d08bee71f60eaab" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.819705 5190 raft_consensus.cc:482] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:39:37.821049 4740 leader_election.cc:341] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.821498 4740 leader_election.cc:302] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.816624 5104 raft_consensus.cc:482] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.829106 5198 raft_consensus.cc:2726] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.821519 5190 raft_consensus.cc:504] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.820612 4775 raft_consensus.cc:2454] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:37.832477 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:37.834538 4737 leader_election.cc:341] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.831876 5190 leader_election.cc:288] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.844658 4664 leader_election.cc:302] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.829166 5104 raft_consensus.cc:504] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.847829 5130 consensus_queue.cc:1035] T 9e05551863b345f78f90e88220cbb163 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:37.849728 5104 leader_election.cc:288] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.857698 5110 raft_consensus.cc:504] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.854097 5071 raft_consensus.cc:2781] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:37.859449 5071 raft_consensus.cc:482] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:37.859889 5071 raft_consensus.cc:3037] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.860635 5110 leader_election.cc:288] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.860529 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:37.869064 4895 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:37.869583 4895 ts_tablet_manager.cc:1367] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.013s
I20240502 21:39:37.869585 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.872628 4895 raft_consensus.cc:348] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.873330 4895 raft_consensus.cc:374] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.873620 4895 raft_consensus.cc:729] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.882692 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:37.883391 4694 raft_consensus.cc:3037] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:37.876710 4817 leader_election.cc:341] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:37.884243 4818 leader_election.cc:341] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.885974 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c52402c2811e45eb91e35a60c0fb0701" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:37.886293 4818 leader_election.cc:302] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:37.886673 4771 raft_consensus.cc:3037] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.887562 5112 raft_consensus.cc:2726] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.877207 4895 consensus_queue.cc:260] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.891101 4851 raft_consensus.cc:3037] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.892514 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.893733 4896 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:37.894196 4896 ts_tablet_manager.cc:1367] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.137s user 0.013s sys 0.012s
W20240502 21:39:37.895210 4740 leader_election.cc:341] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.895637 4740 leader_election.cc:302] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:37.901448 5198 raft_consensus.cc:2726] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.901532 4696 raft_consensus.cc:3037] T 9166e2d9377e4adb8fdbb58e8641e2a1 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:37.889954 5130 consensus_queue.cc:1035] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240502 21:39:37.902370 4694 raft_consensus.cc:2454] T c52402c2811e45eb91e35a60c0fb0701 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.904016 4896 raft_consensus.cc:348] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.904805 4896 raft_consensus.cc:374] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:37.905134 4896 raft_consensus.cc:729] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:37.904537 5104 raft_consensus.cc:482] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.905678 5104 raft_consensus.cc:504] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.906005 4818 leader_election.cc:302] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:37.905782 4896 consensus_queue.cc:260] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.906816 5112 raft_consensus.cc:2781] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:37.907294 5112 raft_consensus.cc:686] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:37.908097 5112 consensus_queue.cc:237] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.909096 5104 leader_election.cc:288] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.910962 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:37.917384 4817 leader_election.cc:341] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.919332 4896 ts_tablet_manager.cc:1398] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s
I20240502 21:39:37.920840 4896 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:37.933300 5104 raft_consensus.cc:482] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.934401 4896 tablet_bootstrap.cc:654] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.933928 5104 raft_consensus.cc:504] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.936807 4895 ts_tablet_manager.cc:1398] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.067s user 0.006s sys 0.002s
I20240502 21:39:37.937230 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
W20240502 21:39:37.939033 4818 leader_election.cc:341] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.939529 4818 leader_election.cc:302] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:37.941237 4899 tablet_bootstrap.cc:654] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:37.941166 5110 raft_consensus.cc:2726] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:37.945657 4895 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:37.946882 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:37.952031 4662 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 58286.6us
I20240502 21:39:37.953163 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fda9381a074590aec89148d7b517cf" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:37.941262 5104 leader_election.cc:288] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:37.957991 4771 raft_consensus.cc:2454] T c52402c2811e45eb91e35a60c0fb0701 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:37.955487 5125 consensus_queue.cc:1035] T f0fda9381a074590aec89148d7b517cf P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240502 21:39:37.963376 4818 leader_election.cc:302] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:37.968698 5104 raft_consensus.cc:2726] T f0fda9381a074590aec89148d7b517cf P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:39:37.969316 5071 raft_consensus.cc:504] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.972028 5129 raft_consensus.cc:482] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:37.972539 5129 raft_consensus.cc:504] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:37.974411 5129 leader_election.cc:288] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:37.988772 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
W20240502 21:39:37.991118 4662 leader_election.cc:341] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:37.979218 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.005900 4774 raft_consensus.cc:2454] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:38.007473 5071 leader_election.cc:288] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.009908 4664 leader_election.cc:302] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters: df8e5a877b6640d692917083c9fb5282
I20240502 21:39:38.017994 5130 raft_consensus.cc:482] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.018558 5130 raft_consensus.cc:504] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.017827 4594 catalog_manager.cc:5484] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:38.020752 5129 raft_consensus.cc:2781] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.022821 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a911c9636f426e9d08bee71f60eaab" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:38.023778 4852 raft_consensus.cc:3037] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.024582 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a911c9636f426e9d08bee71f60eaab" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.023223 5129 raft_consensus.cc:482] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.025233 4774 raft_consensus.cc:3037] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.028003 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.026993 5129 raft_consensus.cc:3037] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.060554 4899 tablet_bootstrap.cc:492] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.061343 4899 ts_tablet_manager.cc:1367] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.244s user 0.021s sys 0.016s
W20240502 21:39:38.062969 4664 leader_election.cc:341] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.064056 4852 raft_consensus.cc:2454] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:38.049714 5130 leader_election.cc:288] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.068260 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.068975 4662 leader_election.cc:302] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.068974 4850 raft_consensus.cc:2454] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 0.
I20240502 21:39:38.075521 4662 leader_election.cc:302] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:38.084957 5256 raft_consensus.cc:2781] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.086771 5256 raft_consensus.cc:482] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.087246 5256 raft_consensus.cc:3037] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.081197 5125 raft_consensus.cc:2781] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.088264 5125 raft_consensus.cc:686] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:38.088749 4896 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:38.089262 4896 ts_tablet_manager.cc:1367] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.169s user 0.017s sys 0.000s
I20240502 21:39:38.089188 5125 consensus_queue.cc:237] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.092999 4895 tablet_bootstrap.cc:654] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.092043 4896 raft_consensus.cc:348] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.095460 4896 raft_consensus.cc:374] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.095853 4896 raft_consensus.cc:729] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.101051 4896 consensus_queue.cc:260] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.099414 5104 consensus_queue.cc:1035] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240502 21:39:38.119550 4899 raft_consensus.cc:348] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.120755 4899 raft_consensus.cc:374] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.121104 4899 raft_consensus.cc:729] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.125013 4899 consensus_queue.cc:260] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.128818 4896 ts_tablet_manager.cc:1398] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20240502 21:39:38.133170 4896 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:38.110760 4774 raft_consensus.cc:2454] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:38.142203 4899 ts_tablet_manager.cc:1398] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.080s user 0.006s sys 0.001s
I20240502 21:39:38.159189 4895 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
W20240502 21:39:38.219951 4818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 47697.1us
I20240502 21:39:38.195314 5256 raft_consensus.cc:504] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.285214 4993 raft_consensus.cc:482] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.285810 4993 raft_consensus.cc:504] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.287475 5256 leader_election.cc:288] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.277215 5129 raft_consensus.cc:504] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.277302 4895 ts_tablet_manager.cc:1367] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.335s user 0.030s sys 0.041s
I20240502 21:39:38.289533 5129 leader_election.cc:288] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Requested vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.290756 4895 raft_consensus.cc:348] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.291572 4895 raft_consensus.cc:374] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.291877 4895 raft_consensus.cc:729] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.292507 4895 consensus_queue.cc:260] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.294744 4895 ts_tablet_manager.cc:1398] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:39:38.295694 4895 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.276691 4899 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.273619 4896 tablet_bootstrap.cc:654] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.301846 4899 tablet_bootstrap.cc:654] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.287791 4993 leader_election.cc:288] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.304760 4849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:38.305397 4849 raft_consensus.cc:3037] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:38.307572 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:38.309998 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.311417 4849 raft_consensus.cc:2454] T ed2181a7e8194705934dadd2d4303b8a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:38.316110 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:38.318769 5112 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:38.340075 4899 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.340839 4899 ts_tablet_manager.cc:1367] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.000s
I20240502 21:39:38.344362 4899 raft_consensus.cc:348] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.345374 4899 raft_consensus.cc:374] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.345732 4899 raft_consensus.cc:729] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.346511 4899 consensus_queue.cc:260] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.351253 4895 tablet_bootstrap.cc:654] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.359145 4899 ts_tablet_manager.cc:1398] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240502 21:39:38.364699 4899 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.367120 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.367820 4770 raft_consensus.cc:3037] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.372964 4899 tablet_bootstrap.cc:654] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.375871 4770 raft_consensus.cc:2454] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:38.378599 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.386538 4774 raft_consensus.cc:3037] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.394097 5265 raft_consensus.cc:482] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.398886 4896 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:38.399437 4896 ts_tablet_manager.cc:1367] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.270s user 0.018s sys 0.022s
I20240502 21:39:38.396234 4774 raft_consensus.cc:2454] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09ff1ee782864ac99d05a7f011aeef44 in term 1.
I20240502 21:39:38.394670 5265 raft_consensus.cc:504] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.410085 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2181a7e8194705934dadd2d4303b8a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.410799 5265 leader_election.cc:288] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.410797 4698 raft_consensus.cc:2379] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:38.415530 4594 catalog_manager.cc:5484] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:38.417582 4664 leader_election.cc:302] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:38.418411 5264 raft_consensus.cc:2781] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.418557 4664 leader_election.cc:302] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545; no voters:
I20240502 21:39:38.418921 5264 raft_consensus.cc:686] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:38.420322 4895 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.420593 5256 raft_consensus.cc:2781] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.419778 5264 consensus_queue.cc:237] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.420817 4895 ts_tablet_manager.cc:1367] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.125s user 0.012s sys 0.011s
I20240502 21:39:38.425132 5256 raft_consensus.cc:686] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Becoming Leader. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Running, Role: LEADER
I20240502 21:39:38.424144 5112 consensus_queue.cc:1035] T 67a2fb6c68b641a0a4e1ab5356b23c3d P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:38.426090 5256 consensus_queue.cc:237] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.423811 5110 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 21:39:38.427637 4895 raft_consensus.cc:348] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.428391 4895 raft_consensus.cc:374] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.428705 4895 raft_consensus.cc:729] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.429786 4895 consensus_queue.cc:260] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:38.421139 4662 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 61113.1us
W20240502 21:39:38.431495 4662 leader_election.cc:341] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.432332 4895 ts_tablet_manager.cc:1398] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:39:38.433588 4895 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.433611 5112 consensus_queue.cc:1035] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:38.436594 4896 raft_consensus.cc:348] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.442077 4896 raft_consensus.cc:374] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.442909 4896 raft_consensus.cc:729] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.442612 4899 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.446041 4899 ts_tablet_manager.cc:1367] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.005s
I20240502 21:39:38.443285 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.439903 4895 tablet_bootstrap.cc:654] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
W20240502 21:39:38.454655 4662 leader_election.cc:341] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.458694 5112 raft_consensus.cc:482] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.458592 4740 leader_election.cc:302] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:38.459311 5112 raft_consensus.cc:504] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.459636 4993 raft_consensus.cc:2726] T ed2181a7e8194705934dadd2d4303b8a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:38.445339 4896 consensus_queue.cc:260] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.461954 5112 leader_election.cc:288] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.463388 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.464164 4896 ts_tablet_manager.cc:1398] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.064s user 0.004s sys 0.004s
I20240502 21:39:38.464087 4774 raft_consensus.cc:2454] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:38.465370 4817 leader_election.cc:302] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.465623 4895 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.466441 5110 raft_consensus.cc:2781] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.466812 5110 raft_consensus.cc:482] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.467190 5110 raft_consensus.cc:3037] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.466964 4895 ts_tablet_manager.cc:1367] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.004s
I20240502 21:39:38.471299 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.472852 4895 raft_consensus.cc:348] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.473598 4895 raft_consensus.cc:374] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.473881 4895 raft_consensus.cc:729] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.474560 4895 consensus_queue.cc:260] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.474833 5110 raft_consensus.cc:504] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:38.475991 4818 leader_election.cc:341] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.478914 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.479566 4773 raft_consensus.cc:3037] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.479674 4895 ts_tablet_manager.cc:1398] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 21:39:38.481699 4895 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.455842 4899 raft_consensus.cc:348] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.485906 4899 raft_consensus.cc:374] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.486215 4899 raft_consensus.cc:729] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.487407 4773 raft_consensus.cc:2454] T 5fb1726ee5764d42806fc4c077226bf9 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:38.487599 4899 consensus_queue.cc:260] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.488591 4817 leader_election.cc:302] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.489290 5104 raft_consensus.cc:2781] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.489774 5104 raft_consensus.cc:686] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:38.489898 4895 tablet_bootstrap.cc:654] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.490638 5104 consensus_queue.cc:237] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.476766 5110 leader_election.cc:288] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.502328 4895 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.502969 4895 ts_tablet_manager.cc:1367] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240502 21:39:38.504088 4896 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:38.506150 4895 raft_consensus.cc:348] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.506522 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb1726ee5764d42806fc4c077226bf9" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:38.506855 4895 raft_consensus.cc:374] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.507184 4895 raft_consensus.cc:729] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.507648 5110 consensus_queue.cc:1035] T c52402c2811e45eb91e35a60c0fb0701 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:38.507833 4895 consensus_queue.cc:260] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:38.509527 4818 leader_election.cc:341] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.510124 4895 ts_tablet_manager.cc:1398] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:39:38.511054 4895 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.511616 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.530962 5272 raft_consensus.cc:482] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.531787 5272 raft_consensus.cc:504] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.571435 4895 tablet_bootstrap.cc:654] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.578537 4594 catalog_manager.cc:5484] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:38.580621 4594 catalog_manager.cc:5484] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193). New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:38.587709 4899 ts_tablet_manager.cc:1398] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.134s user 0.005s sys 0.000s
I20240502 21:39:38.596510 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.596769 4593 catalog_manager.cc:5484] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
W20240502 21:39:38.598205 4662 leader_election.cc:341] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.598881 5272 leader_election.cc:288] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.603596 4899 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.611742 4895 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.613858 4899 tablet_bootstrap.cc:654] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.614477 4895 ts_tablet_manager.cc:1367] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.104s user 0.028s sys 0.024s
I20240502 21:39:38.618294 4895 raft_consensus.cc:348] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.619228 4895 raft_consensus.cc:374] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.619611 4895 raft_consensus.cc:729] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.624069 4899 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.624701 4899 ts_tablet_manager.cc:1367] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.007s
I20240502 21:39:38.624727 4895 consensus_queue.cc:260] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.627050 4896 tablet_bootstrap.cc:654] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.628351 4895 ts_tablet_manager.cc:1398] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240502 21:39:38.630900 4899 raft_consensus.cc:348] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.632040 4899 raft_consensus.cc:374] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.632411 4899 raft_consensus.cc:729] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.633112 4899 consensus_queue.cc:260] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.635771 4899 ts_tablet_manager.cc:1398] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20240502 21:39:38.636750 4899 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.638799 5104 raft_consensus.cc:482] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.638834 4895 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.639432 5104 raft_consensus.cc:504] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.641887 5104 leader_election.cc:288] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.643463 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c974ff399fa4517af1e6d06b99af4f5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.644212 4770 raft_consensus.cc:2454] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:38.645279 4817 leader_election.cc:302] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.647084 5104 raft_consensus.cc:2781] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.647472 5104 raft_consensus.cc:482] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.647821 5104 raft_consensus.cc:3037] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.648679 5252 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:38.651818 4899 tablet_bootstrap.cc:654] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.647776 4895 tablet_bootstrap.cc:654] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.656873 5104 raft_consensus.cc:504] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.658730 5104 leader_election.cc:288] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.660826 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c974ff399fa4517af1e6d06b99af4f5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.661484 4770 raft_consensus.cc:3037] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.670138 4770 raft_consensus.cc:2454] T 9c974ff399fa4517af1e6d06b99af4f5 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:38.671216 4817 leader_election.cc:302] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.671960 5110 raft_consensus.cc:2781] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.672475 5110 raft_consensus.cc:686] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:38.673291 5110 consensus_queue.cc:237] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.691313 5104 raft_consensus.cc:482] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.691838 5104 raft_consensus.cc:504] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.693934 5104 leader_election.cc:288] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.695189 4899 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.695688 4899 ts_tablet_manager.cc:1367] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.003s
I20240502 21:39:38.696658 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.700171 5284 raft_consensus.cc:482] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.699919 4899 raft_consensus.cc:348] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.700955 4899 raft_consensus.cc:374] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.700757 5284 raft_consensus.cc:504] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.701318 4899 raft_consensus.cc:729] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.703500 5284 leader_election.cc:288] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.702421 4899 consensus_queue.cc:260] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.705330 5198 raft_consensus.cc:482] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.705437 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.706024 4899 ts_tablet_manager.cc:1398] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 21:39:38.705803 5198 raft_consensus.cc:504] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:38.706861 4737 leader_election.cc:341] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.706898 4899 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.707700 5198 leader_election.cc:288] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:38.708825 4817 leader_election.cc:341] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.709008 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:38.709595 4852 raft_consensus.cc:2454] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:38.709769 4895 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.710345 4895 ts_tablet_manager.cc:1367] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.072s user 0.023s sys 0.027s
I20240502 21:39:38.710636 4737 leader_election.cc:302] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.711436 5198 raft_consensus.cc:2781] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.711910 5198 raft_consensus.cc:482] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.712241 5198 raft_consensus.cc:3037] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.714357 4899 tablet_bootstrap.cc:654] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.714109 4895 raft_consensus.cc:348] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.715416 4895 raft_consensus.cc:374] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.715857 4895 raft_consensus.cc:729] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.716971 4895 consensus_queue.cc:260] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.719483 5198 raft_consensus.cc:504] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.721107 4895 ts_tablet_manager.cc:1398] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 21:39:38.728183 4895 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.737531 4895 tablet_bootstrap.cc:654] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.747038 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:38.725448 4664 leader_election.cc:341] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.753010 4664 leader_election.cc:302] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:38.754046 5261 raft_consensus.cc:2726] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:38.758359 4895 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.759042 4895 ts_tablet_manager.cc:1367] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.003s
I20240502 21:39:38.762521 4895 raft_consensus.cc:348] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.763470 4895 raft_consensus.cc:374] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.763921 4895 raft_consensus.cc:729] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.774513 4895 consensus_queue.cc:260] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.775626 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:38.776587 4851 raft_consensus.cc:3037] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.777827 4895 ts_tablet_manager.cc:1398] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20240502 21:39:38.779733 4895 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.787279 4895 tablet_bootstrap.cc:654] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.802011 4895 tablet_bootstrap.cc:492] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.802516 4895 ts_tablet_manager.cc:1367] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.008s
I20240502 21:39:38.804986 4895 raft_consensus.cc:348] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.805866 4895 raft_consensus.cc:374] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.806136 4895 raft_consensus.cc:729] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.806998 4895 consensus_queue.cc:260] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.809625 4895 ts_tablet_manager.cc:1398] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240502 21:39:38.810526 4895 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.817801 4895 tablet_bootstrap.cc:654] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.773818 5198 leader_election.cc:288] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:38.828122 4895 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.828641 4895 ts_tablet_manager.cc:1367] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20240502 21:39:38.829866 4851 raft_consensus.cc:2454] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:38.831084 4737 leader_election.cc:302] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.831929 5198 raft_consensus.cc:2781] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:38.834285 4594 catalog_manager.cc:5484] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:38.835250 4895 raft_consensus.cc:348] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.836120 4895 raft_consensus.cc:374] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.836222 5198 raft_consensus.cc:686] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:38.836459 4895 raft_consensus.cc:729] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.837545 5198 consensus_queue.cc:237] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.837320 5178 raft_consensus.cc:482] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.845981 4895 consensus_queue.cc:260] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.845382 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.846506 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.846318 5178 raft_consensus.cc:504] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.848503 4895 ts_tablet_manager.cc:1398] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240502 21:39:38.849642 4895 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282: Bootstrap starting.
I20240502 21:39:38.850395 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e028b61d3e4a92b66697049b88e05a" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:38.851778 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.850759 5178 leader_election.cc:288] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:38.853574 4817 leader_election.cc:341] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.855738 4899 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.856279 4896 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:38.856703 4899 ts_tablet_manager.cc:1367] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.150s user 0.053s sys 0.078s
I20240502 21:39:38.856793 4895 tablet_bootstrap.cc:654] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.856719 5178 raft_consensus.cc:482] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.857573 5178 raft_consensus.cc:504] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.859670 4899 raft_consensus.cc:348] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.860464 4899 raft_consensus.cc:374] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.860599 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.860826 4899 raft_consensus.cc:729] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.861433 4773 raft_consensus.cc:2379] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df8e5a877b6640d692917083c9fb5282 in current term 1: Already voted for candidate 09ff1ee782864ac99d05a7f011aeef44 in this term.
I20240502 21:39:38.861583 4899 consensus_queue.cc:260] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.862766 5178 leader_election.cc:288] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.864588 4899 ts_tablet_manager.cc:1398] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240502 21:39:38.856762 4896 ts_tablet_manager.cc:1367] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.392s user 0.009s sys 0.002s
I20240502 21:39:38.869616 4899 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:38.868813 4896 raft_consensus.cc:348] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.870182 4896 raft_consensus.cc:374] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.870476 4896 raft_consensus.cc:729] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.872849 4593 catalog_manager.cc:5484] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:38.876044 4895 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282: No bootstrap required, opened a new log
I20240502 21:39:38.876523 4899 tablet_bootstrap.cc:654] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.876556 4895 ts_tablet_manager.cc:1367] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.007s
I20240502 21:39:38.878468 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c974ff399fa4517af1e6d06b99af4f5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.879168 4895 raft_consensus.cc:348] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.879878 4895 raft_consensus.cc:374] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:38.880158 4895 raft_consensus.cc:729] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Initialized, Role: FOLLOWER
I20240502 21:39:38.880117 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c974ff399fa4517af1e6d06b99af4f5" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:38.881101 4895 consensus_queue.cc:260] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.881721 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.883414 4895 ts_tablet_manager.cc:1398] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:39:38.871224 4896 consensus_queue.cc:260] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.886492 4896 ts_tablet_manager.cc:1398] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240502 21:39:38.891772 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.892762 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14af5e6ab984917b771589cdb45a779" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.896924 4951 raft_consensus.cc:482] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:38.897598 4951 raft_consensus.cc:504] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.899567 4951 leader_election.cc:288] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.901221 4896 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:38.902263 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:38.902945 4770 raft_consensus.cc:2454] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:38.903980 4817 leader_election.cc:302] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:38.904960 5178 raft_consensus.cc:2781] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:38.905326 5178 raft_consensus.cc:482] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:38.905710 5178 raft_consensus.cc:3037] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:38.907572 4896 tablet_bootstrap.cc:654] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:38.912873 5178 raft_consensus.cc:504] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.915232 5178 leader_election.cc:288] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:38.916369 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:38.917227 4773 raft_consensus.cc:3037] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:38.923606 4818 leader_election.cc:341] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:38.924592 4818 leader_election.cc:341] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:38.925343 4818 leader_election.cc:341] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.925733 4818 leader_election.cc:302] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:38.926422 5112 raft_consensus.cc:2726] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:39:38.931380 4740 leader_election.cc:341] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:38.932265 4740 leader_election.cc:341] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.933699 4740 leader_election.cc:302] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:38.934384 5284 raft_consensus.cc:2726] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:39:38.938722 4664 leader_election.cc:341] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:38.938722 4740 leader_election.cc:341] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.939291 4664 leader_election.cc:302] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:38.942517 5261 raft_consensus.cc:2726] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:38.948432 5265 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:38.926517 4773 raft_consensus.cc:2454] T 3a0e4a551d0f4f058d2d97346f8a247b P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:38.955502 4896 tablet_bootstrap.cc:492] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:38.957049 4896 ts_tablet_manager.cc:1367] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.001s
I20240502 21:39:38.960985 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:38.961726 4694 raft_consensus.cc:2454] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:38.962011 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0e4a551d0f4f058d2d97346f8a247b" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:38.962668 4699 raft_consensus.cc:3037] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:38.963006 4818 leader_election.cc:341] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:38.963461 4818 leader_election.cc:302] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:38.964610 4818 leader_election.cc:302] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:38.994166 4899 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:38.994719 4899 ts_tablet_manager.cc:1367] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.125s user 0.007s sys 0.023s
I20240502 21:39:38.995375 5104 raft_consensus.cc:3037] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.020488 5265 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:39.020824 4899 raft_consensus.cc:348] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.021574 4899 raft_consensus.cc:374] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.021850 4899 raft_consensus.cc:729] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.022406 5178 raft_consensus.cc:482] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.022469 4899 consensus_queue.cc:260] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.023015 5178 raft_consensus.cc:504] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:38.996428 4951 raft_consensus.cc:2726] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:39.026563 4899 ts_tablet_manager.cc:1398] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20240502 21:39:39.027451 4899 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
W20240502 21:39:39.035622 4818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 70030.9us
I20240502 21:39:39.038064 4699 raft_consensus.cc:2454] T 3a0e4a551d0f4f058d2d97346f8a247b P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.047940 4817 leader_election.cc:302] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.048638 4951 raft_consensus.cc:2781] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.049095 4951 raft_consensus.cc:686] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:39.049876 4951 consensus_queue.cc:237] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.060329 5110 raft_consensus.cc:482] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.060752 5104 raft_consensus.cc:2726] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:39.047788 5302 raft_consensus.cc:482] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.079144 5302 raft_consensus.cc:504] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.074460 5190 raft_consensus.cc:482] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.061004 5110 raft_consensus.cc:504] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.071568 5252 consensus_queue.cc:1035] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240502 21:39:39.086108 5190 raft_consensus.cc:504] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.087762 5110 leader_election.cc:288] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.088351 5265 consensus_queue.cc:1035] T ed2181a7e8194705934dadd2d4303b8a P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:39:39.092661 4818 consensus_peers.cc:485] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:39.101543 4896 raft_consensus.cc:348] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.103603 4896 raft_consensus.cc:374] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.104138 4896 raft_consensus.cc:729] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.088096 5190 leader_election.cc:288] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.112273 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.112787 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.113348 4699 raft_consensus.cc:2454] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:39.114018 4696 raft_consensus.cc:2454] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:39.117398 5302 leader_election.cc:288] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.186620 4899 tablet_bootstrap.cc:654] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.138355 4818 leader_election.cc:302] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.213274 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
W20240502 21:39:39.215672 4817 leader_election.cc:341] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:39.216177 4818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 77972.7us
I20240502 21:39:39.216361 4951 raft_consensus.cc:2781] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.216742 4951 raft_consensus.cc:482] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.216863 4818 leader_election.cc:302] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.217082 4951 raft_consensus.cc:3037] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:39:39.213481 5301 log.cc:924] Time spent T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545: Append to log took a long time: real 0.111s user 0.010s sys 0.009s
I20240502 21:39:39.220326 5306 raft_consensus.cc:2781] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.145105 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.231133 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.147142 5266 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.085s
I20240502 21:39:39.234496 4592 catalog_manager.cc:5484] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:39.238880 4951 raft_consensus.cc:504] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
W20240502 21:39:39.247547 4818 leader_election.cc:341] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.247994 4818 leader_election.cc:302] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44, ddea410051614f23b93c7342ec7cc545
I20240502 21:39:39.140522 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:39.240689 5306 raft_consensus.cc:482] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.245841 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:39.252954 4850 raft_consensus.cc:2454] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:39.242175 4896 consensus_queue.cc:260] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.161273 5178 leader_election.cc:288] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.254117 5303 raft_consensus.cc:2726] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:39.241744 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:39.256484 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.259027 4951 leader_election.cc:288] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
W20240502 21:39:39.262037 4740 leader_election.cc:341] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.254443 5306 raft_consensus.cc:3037] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.273545 4896 ts_tablet_manager.cc:1398] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.316s user 0.013s sys 0.003s
I20240502 21:39:39.266381 4699 raft_consensus.cc:3037] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.256376 4770 raft_consensus.cc:2454] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:39.289248 4896 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
W20240502 21:39:39.304647 4737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 80645.3us
W20240502 21:39:39.309296 4817 leader_election.cc:341] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.315021 4951 raft_consensus.cc:482] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.315459 4951 raft_consensus.cc:504] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.317780 4951 leader_election.cc:288] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.338910 4899 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:39.339486 4899 ts_tablet_manager.cc:1367] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.312s user 0.014s sys 0.012s
I20240502 21:39:39.349433 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:39.363761 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:39.371966 4737 leader_election.cc:302] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters: 09ff1ee782864ac99d05a7f011aeef44
W20240502 21:39:39.375358 4644 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240502 21:39:39.383889 4951 consensus_queue.cc:1035] T 5fb1726ee5764d42806fc4c077226bf9 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:39:39.384246 4817 leader_election.cc:341] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.384526 5303 consensus_queue.cc:1035] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 21:39:39.376613 5313 raft_consensus.cc:2781] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.382823 5283 raft_consensus.cc:482] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.390892 5313 raft_consensus.cc:482] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.391289 5313 raft_consensus.cc:3037] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.391116 5283 raft_consensus.cc:504] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.393162 5283 leader_election.cc:288] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
W20240502 21:39:39.394901 4818 consensus_peers.cc:485] T 9c974ff399fa4517af1e6d06b99af4f5 P df8e5a877b6640d692917083c9fb5282 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:39.391973 4899 raft_consensus.cc:348] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.398828 4899 raft_consensus.cc:374] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.374855 4771 raft_consensus.cc:2454] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:39.399268 4899 raft_consensus.cc:729] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.400735 4899 consensus_queue.cc:260] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.399600 4699 raft_consensus.cc:2454] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.413484 5306 raft_consensus.cc:504] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.415653 4899 ts_tablet_manager.cc:1398] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.076s user 0.008s sys 0.000s
I20240502 21:39:39.416656 4899 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545: Bootstrap starting.
I20240502 21:39:39.417534 5313 raft_consensus.cc:504] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.415699 5306 leader_election.cc:288] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.413405 4896 tablet_bootstrap.cc:654] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.419796 5313 leader_election.cc:288] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.422540 4899 tablet_bootstrap.cc:654] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.431668 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:39.432425 4852 raft_consensus.cc:2454] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:39.434958 4817 leader_election.cc:302] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.438639 5306 raft_consensus.cc:2781] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.439018 5306 raft_consensus.cc:482] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.439348 5306 raft_consensus.cc:3037] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.452729 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.453691 4699 raft_consensus.cc:2454] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 0.
I20240502 21:39:39.455111 5322 raft_consensus.cc:482] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.455586 5322 raft_consensus.cc:504] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.457928 5306 raft_consensus.cc:504] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.462087 4775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:39.462735 4775 raft_consensus.cc:3037] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.465157 5322 leader_election.cc:288] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.469723 4775 raft_consensus.cc:2454] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.473888 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c65cfc531344398b70ce5862b2211fd" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.474591 4697 raft_consensus.cc:3037] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.475494 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:39.476126 4848 raft_consensus.cc:3037] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.486176 4852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:39.486871 4852 raft_consensus.cc:2454] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 0.
I20240502 21:39:39.488749 4737 leader_election.cc:302] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.488932 4817 leader_election.cc:302] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.489501 5322 raft_consensus.cc:2781] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.489697 5178 raft_consensus.cc:2781] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.490443 5178 raft_consensus.cc:686] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:39.491245 5178 consensus_queue.cc:237] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.489885 5322 raft_consensus.cc:482] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.495496 5322 raft_consensus.cc:3037] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.495512 4848 raft_consensus.cc:2454] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:39.497185 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.497907 4694 raft_consensus.cc:3037] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.498006 4896 tablet_bootstrap.cc:492] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:39.500458 4896 ts_tablet_manager.cc:1367] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.013s
I20240502 21:39:39.489687 4737 leader_election.cc:302] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.512506 5313 raft_consensus.cc:2781] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:39:39.513115 5313 raft_consensus.cc:482] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:39:39.513537 5313 raft_consensus.cc:3037] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.512295 4896 raft_consensus.cc:348] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.514585 4896 raft_consensus.cc:374] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.514946 4896 raft_consensus.cc:729] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.517396 4737 leader_election.cc:302] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.521885 5322 raft_consensus.cc:504] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.503759 4697 raft_consensus.cc:2454] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.514822 4694 raft_consensus.cc:2454] T a06147be82454bb0a23d0f3e68c06f04 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.526229 5322 leader_election.cc:288] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.515676 4896 consensus_queue.cc:260] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.530225 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:39.530681 4896 ts_tablet_manager.cc:1398] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20240502 21:39:39.530921 4850 raft_consensus.cc:3037] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.531598 5312 raft_consensus.cc:2781] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.532752 5312 raft_consensus.cc:686] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:39.532931 5306 leader_election.cc:288] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:39.533655 5312 consensus_queue.cc:237] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.534315 4896 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:39.539187 4591 catalog_manager.cc:5484] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:39.542223 5313 raft_consensus.cc:504] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.544122 4850 raft_consensus.cc:2454] T d296f3d569f143fca949d49feb27d19d P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:39.554179 5313 leader_election.cc:288] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Requested vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.557416 5322 consensus_queue.cc:1035] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:39.558240 4737 leader_election.cc:302] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.559749 5321 raft_consensus.cc:2781] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.559955 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.562276 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282"
I20240502 21:39:39.562418 5321 raft_consensus.cc:686] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:39.565106 5321 consensus_queue.cc:237] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.568136 4850 raft_consensus.cc:3037] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.583273 4896 tablet_bootstrap.cc:654] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.584908 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06147be82454bb0a23d0f3e68c06f04" candidate_uuid: "df8e5a877b6640d692917083c9fb5282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545"
I20240502 21:39:39.585652 4771 raft_consensus.cc:3037] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.585496 4697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.586162 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.588771 4818 leader_election.cc:302] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters: ddea410051614f23b93c7342ec7cc545
I20240502 21:39:39.598011 4850 raft_consensus.cc:2454] T bc0f282d043b43e8b9548ec320bdad33 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddea410051614f23b93c7342ec7cc545 in term 1.
I20240502 21:39:39.599938 4737 leader_election.cc:302] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.605818 4899 tablet_bootstrap.cc:492] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545: No bootstrap required, opened a new log
I20240502 21:39:39.606415 4899 ts_tablet_manager.cc:1367] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545: Time spent bootstrapping tablet: real 0.190s user 0.029s sys 0.009s
W20240502 21:39:39.607751 4740 leader_election.cc:341] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.608044 4818 leader_election.cc:302] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282; no voters:
I20240502 21:39:39.609452 5178 raft_consensus.cc:2781] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.609875 5178 raft_consensus.cc:686] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:39.610590 5178 consensus_queue.cc:237] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.614427 5104 consensus_queue.cc:1035] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:39.601897 5313 raft_consensus.cc:2781] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.600714 5306 raft_consensus.cc:2781] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:39:39.619024 4693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.622179 5313 raft_consensus.cc:686] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Becoming Leader. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Running, Role: LEADER
I20240502 21:39:39.623092 5313 consensus_queue.cc:237] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.630151 5306 raft_consensus.cc:686] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Becoming Leader. State: Replica: df8e5a877b6640d692917083c9fb5282, State: Running, Role: LEADER
I20240502 21:39:39.639799 5306 consensus_queue.cc:237] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.631714 4698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0f282d043b43e8b9548ec320bdad33" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44"
I20240502 21:39:39.651197 4594 catalog_manager.cc:5484] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
W20240502 21:39:39.663625 4740 consensus_peers.cc:485] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:39.667716 4771 raft_consensus.cc:2454] T a06147be82454bb0a23d0f3e68c06f04 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df8e5a877b6640d692917083c9fb5282 in term 1.
I20240502 21:39:39.670534 4899 raft_consensus.cc:348] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.671376 4899 raft_consensus.cc:374] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.671643 4899 raft_consensus.cc:729] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddea410051614f23b93c7342ec7cc545, State: Initialized, Role: FOLLOWER
W20240502 21:39:39.673743 4740 leader_election.cc:341] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:39.675220 4740 leader_election.cc:341] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:39:39.679618 4740 leader_election.cc:341] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.677754 4899 consensus_queue.cc:260] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.713750 4899 ts_tablet_manager.cc:1398] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545: Time spent starting tablet: real 0.107s user 0.006s sys 0.000s
I20240502 21:39:39.717042 5110 consensus_queue.cc:1035] T 3a0e4a551d0f4f058d2d97346f8a247b P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:39:39.720026 4740 leader_election.cc:341] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:39:39.727061 4896 tablet_bootstrap.cc:492] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:39.727563 4896 ts_tablet_manager.cc:1367] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.193s user 0.017s sys 0.001s
I20240502 21:39:39.729084 5313 raft_consensus.cc:482] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:39.730127 4896 raft_consensus.cc:348] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.730665 4896 raft_consensus.cc:374] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.730887 4896 raft_consensus.cc:729] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.731420 4896 consensus_queue.cc:260] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.729638 5313 raft_consensus.cc:504] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.734316 5313 leader_election.cc:288] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:39.735126 4848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:39.735402 4699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3c2c7e0cb241adb79d796752f2d4c6" candidate_uuid: "ddea410051614f23b93c7342ec7cc545" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ff1ee782864ac99d05a7f011aeef44" is_pre_election: true
I20240502 21:39:39.736222 4699 raft_consensus.cc:2379] T 2a3c2c7e0cb241adb79d796752f2d4c6 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ddea410051614f23b93c7342ec7cc545 in current term 1: Already voted for candidate df8e5a877b6640d692917083c9fb5282 in this term.
I20240502 21:39:39.742823 4740 leader_election.cc:302] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddea410051614f23b93c7342ec7cc545; no voters: 09ff1ee782864ac99d05a7f011aeef44, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:39.744138 5198 raft_consensus.cc:3037] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.750427 4693 raft_consensus.cc:3037] T 9c974ff399fa4517af1e6d06b99af4f5 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:39.753082 5198 raft_consensus.cc:2726] T 2a3c2c7e0cb241adb79d796752f2d4c6 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:39.756506 4896 ts_tablet_manager.cc:1398] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20240502 21:39:39.757483 4896 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:39.762055 4896 tablet_bootstrap.cc:654] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.808502 4591 catalog_manager.cc:5484] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:39.813614 4591 catalog_manager.cc:5484] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 reported cstate change: term changed from 0 to 1, leader changed from <none> to df8e5a877b6640d692917083c9fb5282 (127.3.113.195). New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } }
I20240502 21:39:39.842224 4896 tablet_bootstrap.cc:492] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:39.843344 4896 ts_tablet_manager.cc:1367] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.007s
I20240502 21:39:39.862131 4896 raft_consensus.cc:348] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.863322 4896 raft_consensus.cc:374] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.863677 4896 raft_consensus.cc:729] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.864600 4896 consensus_queue.cc:260] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.867200 4896 ts_tablet_manager.cc:1398] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.023s user 0.011s sys 0.000s
I20240502 21:39:39.885187 4896 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:39.903497 5112 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:39.905725 4896 tablet_bootstrap.cc:654] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:39.917532 4594 catalog_manager.cc:5484] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:39.919540 4594 catalog_manager.cc:5484] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddea410051614f23b93c7342ec7cc545 (127.3.113.194). New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:39.925563 4896 tablet_bootstrap.cc:492] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:39.926043 4896 ts_tablet_manager.cc:1367] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.005s
I20240502 21:39:39.928954 4896 raft_consensus.cc:348] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.982527 4896 raft_consensus.cc:374] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:39.982987 4896 raft_consensus.cc:729] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:39.983912 4896 consensus_queue.cc:260] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:39.988703 5110 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:39.990263 4896 ts_tablet_manager.cc:1398] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.064s user 0.011s sys 0.003s
I20240502 21:39:39.992029 4896 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:39.998859 4896 tablet_bootstrap.cc:654] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:40.021625 5322 consensus_queue.cc:1035] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:39:40.024075 4740 consensus_peers.cc:485] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:40.030777 4896 tablet_bootstrap.cc:492] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:40.031337 4896 ts_tablet_manager.cc:1367] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.002s
W20240502 21:39:40.024919 4740 consensus_peers.cc:485] T d296f3d569f143fca949d49feb27d19d P ddea410051614f23b93c7342ec7cc545 -> Peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): Couldn't send request to peer 09ff1ee782864ac99d05a7f011aeef44. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:40.034404 4896 raft_consensus.cc:348] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.035356 4896 raft_consensus.cc:374] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:40.046532 4698 raft_consensus.cc:3037] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:40.035704 4896 raft_consensus.cc:729] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:40.049255 4896 consensus_queue.cc:260] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.051680 4896 ts_tablet_manager.cc:1398] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240502 21:39:40.052764 4896 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Bootstrap starting.
I20240502 21:39:40.069926 5190 consensus_queue.cc:1035] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:40.071027 4896 tablet_bootstrap.cc:654] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Neither blocks nor log segments found. Creating new log.
I20240502 21:39:40.097924 4693 raft_consensus.cc:3037] T bc0f282d043b43e8b9548ec320bdad33 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:40.099781 4896 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: No bootstrap required, opened a new log
I20240502 21:39:40.100267 4896 ts_tablet_manager.cc:1367] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.020s
I20240502 21:39:40.103893 5302 consensus_queue.cc:1035] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:40.107167 4896 raft_consensus.cc:348] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.107990 4896 raft_consensus.cc:374] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:39:40.108444 4896 raft_consensus.cc:729] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff1ee782864ac99d05a7f011aeef44, State: Initialized, Role: FOLLOWER
I20240502 21:39:40.110636 4896 consensus_queue.cc:260] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.112250 5178 consensus_queue.cc:1035] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:40.124878 5302 consensus_queue.cc:1035] T 2a3c2c7e0cb241adb79d796752f2d4c6 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:40.141383 5190 consensus_queue.cc:1035] T bc0f282d043b43e8b9548ec320bdad33 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:40.144099 4896 ts_tablet_manager.cc:1398] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Time spent starting tablet: real 0.043s user 0.002s sys 0.004s
I20240502 21:39:40.159307 5190 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:39:40.176875 5302 consensus_queue.cc:1035] T a06147be82454bb0a23d0f3e68c06f04 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:39:40.198472 5261 raft_consensus.cc:482] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:40.199029 5261 raft_consensus.cc:504] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.201165 5261 leader_election.cc:288] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:40.203289 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:40.204237 4851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2394febdff4bb8bd495e4b8086e412" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:40.206765 4695 raft_consensus.cc:3037] T 5fb1726ee5764d42806fc4c077226bf9 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:40.209932 4664 leader_election.cc:302] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:40.213332 5261 raft_consensus.cc:2726] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:40.241255 5319 raft_consensus.cc:482] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:39:40.242352 5319 raft_consensus.cc:504] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } }
I20240502 21:39:40.246943 5319 leader_election.cc:288] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679), df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:40.247272 4850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8e5a877b6640d692917083c9fb5282" is_pre_election: true
I20240502 21:39:40.255939 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d296f3d569f143fca949d49feb27d19d" candidate_uuid: "09ff1ee782864ac99d05a7f011aeef44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddea410051614f23b93c7342ec7cc545" is_pre_election: true
I20240502 21:39:40.259585 4664 leader_election.cc:302] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09ff1ee782864ac99d05a7f011aeef44; no voters: ddea410051614f23b93c7342ec7cc545, df8e5a877b6640d692917083c9fb5282
I20240502 21:39:40.260936 5261 raft_consensus.cc:2726] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:39:40.330278 4695 raft_consensus.cc:3037] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:40.380985 4695 raft_consensus.cc:3037] T d296f3d569f143fca949d49feb27d19d P 09ff1ee782864ac99d05a7f011aeef44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:39:40.462699 3527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:39:40.491341 5361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:39:40.492579 5362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:40.493956 3527 server_base.cc:1034] running on GCE node
W20240502 21:39:40.497112 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:39:40.498327 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 21:39:40.498610 3527 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 21:39:40.498809 3527 hybrid_clock.cc:648] HybridClock initialized: now 1714685980498792 us; error 0 us; skew 500 ppm
I20240502 21:39:40.501030 3527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:39:40.510219 3527 webserver.cc:460] Webserver started at http://127.3.113.196:44289/ using document root <none> and password file <none>
I20240502 21:39:40.510859 3527 fs_manager.cc:359] Metadata directory not provided
I20240502 21:39:40.511112 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:39:40.511444 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:39:40.512941 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-3-root/instance:
uuid: "6a9bb5562dce4197b3ca1f398c12954c"
format_stamp: "Formatted at 2024-05-02 21:39:40 on dist-test-slave-pddb"
I20240502 21:39:40.518499 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:39:40.522306 5370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:40.523267 3527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:39:40.523581 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-3-root
uuid: "6a9bb5562dce4197b3ca1f398c12954c"
format_stamp: "Formatted at 2024-05-02 21:39:40 on dist-test-slave-pddb"
I20240502 21:39:40.523908 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:39:40.549839 3527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:39:40.551339 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:39:40.553277 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:39:40.557674 3527 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:39:40.558027 3527 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 21:39:40.558465 3527 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:39:40.558738 3527 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:39:40.659132 3527 rpc_server.cc:307] RPC server started. Bound to: 127.3.113.196:35981
I20240502 21:39:40.659190 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.113.196:35981 every 8 connection(s)
I20240502 21:39:40.666203 3527 auto_leader_rebalancer-test.cc:405] add a tserver: 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:40.681893 5434 heartbeater.cc:347] Connected to a master server at 127.3.113.254:39185
I20240502 21:39:40.682361 5434 heartbeater.cc:464] Registering TS with master...
I20240502 21:39:40.683249 5434 heartbeater.cc:511] Master 127.3.113.254:39185 requested a full tablet report, sending...
I20240502 21:39:40.685161 4591 ts_manager.cc:194] Registered new tserver with Master: 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981)
I20240502 21:39:40.686643 4591 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37460
I20240502 21:39:41.521525 4850 consensus_queue.cc:237] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:41.527943 4771 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.529914 4695 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.532884 5441 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:41.534078 5302 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:41.547581 5441 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.548831 4771 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.550246 4695 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:41.554607 4816 consensus_peers.cc:485] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 047156e0297249998c3f2c5362c70432. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:41.569486 4591 catalog_manager.cc:5484] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:41.594717 4695 consensus_queue.cc:237] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:41.603154 4850 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.604692 5319 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:41.605206 4771 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.609570 5319 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:39:41.632032 4663 consensus_peers.cc:485] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 856dc0c130c04f4894884164ae24a053. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:41.635704 5319 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.637223 4771 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.639628 4852 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.653201 4850 consensus_queue.cc:237] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:41.656960 4592 catalog_manager.cc:5484] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:41.664637 4816 consensus_peers.cc:485] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c65cfc531344398b70ce5862b2211fd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:41.666534 4771 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.669147 4695 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.670776 5302 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:41.672528 5104 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:41.689880 5434 heartbeater.cc:503] Master 127.3.113.254:39185 was elected leader, sending a full tablet report...
I20240502 21:39:41.698146 5104 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.706588 4695 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.707329 4770 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.728226 4591 catalog_manager.cc:5484] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:41.755316 4770 consensus_queue.cc:237] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:41.762214 4850 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.763707 5190 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:41.764089 4698 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:41.765810 5462 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:41.772982 5190 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:41.781320 4738 consensus_peers.cc:485] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80c35cadfff54236bbf05e32f945728f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:41.782931 4850 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.785990 4698 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:41.794397 4594 catalog_manager.cc:5484] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:41.963820 5466 ts_tablet_manager.cc:902] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:41.971753 5466 tablet_copy_client.cc:308] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.195:45259
I20240502 21:39:41.985395 4862 tablet_copy_service.cc:140] P df8e5a877b6640d692917083c9fb5282: Received BeginTabletCopySession request for tablet 047156e0297249998c3f2c5362c70432 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:47828)
I20240502 21:39:41.986009 4862 tablet_copy_service.cc:161] P df8e5a877b6640d692917083c9fb5282: Beginning new tablet copy session on tablet 047156e0297249998c3f2c5362c70432 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:47828: session id = 6a9bb5562dce4197b3ca1f398c12954c-047156e0297249998c3f2c5362c70432
I20240502 21:39:41.994051 4862 tablet_copy_source_session.cc:205] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:41.998735 5466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047156e0297249998c3f2c5362c70432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:42.012239 5466 tablet_copy_client.cc:783] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:42.012918 5466 tablet_copy_client.cc:647] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:42.017434 5466 tablet_copy_client.cc:523] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:42.026737 5466 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:42.046382 5466 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:42.047435 5466 tablet_bootstrap.cc:492] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:42.048175 5466 ts_tablet_manager.cc:1367] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.011s
I20240502 21:39:42.050734 5466 raft_consensus.cc:348] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.051651 5466 raft_consensus.cc:729] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:42.052278 5466 consensus_queue.cc:260] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.054997 5466 ts_tablet_manager.cc:1398] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:39:42.057005 4862 tablet_copy_service.cc:342] P df8e5a877b6640d692917083c9fb5282: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-047156e0297249998c3f2c5362c70432 received from {username='slave'} at 127.0.0.1:47828
I20240502 21:39:42.057452 4862 tablet_copy_service.cc:433] P df8e5a877b6640d692917083c9fb5282: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-047156e0297249998c3f2c5362c70432 on tablet 047156e0297249998c3f2c5362c70432 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:42.145785 5466 ts_tablet_manager.cc:902] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:42.155809 5466 tablet_copy_client.cc:308] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.195:45259
I20240502 21:39:42.157485 4862 tablet_copy_service.cc:140] P df8e5a877b6640d692917083c9fb5282: Received BeginTabletCopySession request for tablet 1c65cfc531344398b70ce5862b2211fd from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:47828)
I20240502 21:39:42.157902 4862 tablet_copy_service.cc:161] P df8e5a877b6640d692917083c9fb5282: Beginning new tablet copy session on tablet 1c65cfc531344398b70ce5862b2211fd from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:47828: session id = 6a9bb5562dce4197b3ca1f398c12954c-1c65cfc531344398b70ce5862b2211fd
I20240502 21:39:42.162891 4862 tablet_copy_source_session.cc:205] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:42.165930 5466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c65cfc531344398b70ce5862b2211fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:42.170101 5478 ts_tablet_manager.cc:902] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:42.179337 5480 ts_tablet_manager.cc:902] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:42.199585 5466 tablet_copy_client.cc:783] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:42.200343 5466 tablet_copy_client.cc:647] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:42.201216 5478 tablet_copy_client.cc:308] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.194:38679
I20240502 21:39:42.207108 5466 tablet_copy_client.cc:523] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:42.218341 4786 tablet_copy_service.cc:140] P ddea410051614f23b93c7342ec7cc545: Received BeginTabletCopySession request for tablet 80c35cadfff54236bbf05e32f945728f from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:55718)
I20240502 21:39:42.219452 4786 tablet_copy_service.cc:161] P ddea410051614f23b93c7342ec7cc545: Beginning new tablet copy session on tablet 80c35cadfff54236bbf05e32f945728f from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:55718: session id = 6a9bb5562dce4197b3ca1f398c12954c-80c35cadfff54236bbf05e32f945728f
I20240502 21:39:42.230629 5466 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:42.233409 5480 tablet_copy_client.cc:308] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.193:44335
I20240502 21:39:42.235164 4786 tablet_copy_source_session.cc:205] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:42.241489 5478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c35cadfff54236bbf05e32f945728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:42.248358 4709 tablet_copy_service.cc:140] P 09ff1ee782864ac99d05a7f011aeef44: Received BeginTabletCopySession request for tablet 856dc0c130c04f4894884164ae24a053 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:34930)
I20240502 21:39:42.249045 4709 tablet_copy_service.cc:161] P 09ff1ee782864ac99d05a7f011aeef44: Beginning new tablet copy session on tablet 856dc0c130c04f4894884164ae24a053 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:34930: session id = 6a9bb5562dce4197b3ca1f398c12954c-856dc0c130c04f4894884164ae24a053
I20240502 21:39:42.265650 5478 tablet_copy_client.cc:783] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:42.266510 5478 tablet_copy_client.cc:647] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:42.274309 5478 tablet_copy_client.cc:523] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:42.274506 4709 tablet_copy_source_session.cc:205] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:42.277936 5480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856dc0c130c04f4894884164ae24a053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:42.300078 5480 tablet_copy_client.cc:783] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:42.300813 5480 tablet_copy_client.cc:647] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:42.303722 5466 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:42.305159 5466 tablet_bootstrap.cc:492] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:42.305856 5466 ts_tablet_manager.cc:1367] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.075s user 0.037s sys 0.017s
I20240502 21:39:42.307780 5480 tablet_copy_client.cc:523] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:42.308660 5466 raft_consensus.cc:348] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.309754 5466 raft_consensus.cc:729] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:42.310839 5478 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:42.310431 5466 consensus_queue.cc:260] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.318404 5466 ts_tablet_manager.cc:1398] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.012s user 0.003s sys 0.005s
I20240502 21:39:42.318961 5480 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:42.320605 4862 tablet_copy_service.cc:342] P df8e5a877b6640d692917083c9fb5282: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-1c65cfc531344398b70ce5862b2211fd received from {username='slave'} at 127.0.0.1:47828
I20240502 21:39:42.321074 4862 tablet_copy_service.cc:433] P df8e5a877b6640d692917083c9fb5282: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-1c65cfc531344398b70ce5862b2211fd on tablet 1c65cfc531344398b70ce5862b2211fd with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:42.346020 5478 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:42.346975 5478 tablet_bootstrap.cc:492] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:42.347793 5478 ts_tablet_manager.cc:1367] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.005s
I20240502 21:39:42.350210 5478 raft_consensus.cc:348] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.351004 5478 raft_consensus.cc:729] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:42.351594 5478 consensus_queue.cc:260] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.353865 5478 ts_tablet_manager.cc:1398] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:39:42.356225 4786 tablet_copy_service.cc:342] P ddea410051614f23b93c7342ec7cc545: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-80c35cadfff54236bbf05e32f945728f received from {username='slave'} at 127.0.0.1:55718
I20240502 21:39:42.356654 4786 tablet_copy_service.cc:433] P ddea410051614f23b93c7342ec7cc545: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-80c35cadfff54236bbf05e32f945728f on tablet 80c35cadfff54236bbf05e32f945728f with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:42.361351 5480 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:42.362340 5480 tablet_bootstrap.cc:492] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:42.363071 5480 ts_tablet_manager.cc:1367] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.004s
I20240502 21:39:42.365454 5480 raft_consensus.cc:348] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.366277 5480 raft_consensus.cc:729] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:42.366904 5480 consensus_queue.cc:260] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:42.369176 5480 ts_tablet_manager.cc:1398] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 21:39:42.370693 4709 tablet_copy_service.cc:342] P 09ff1ee782864ac99d05a7f011aeef44: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-856dc0c130c04f4894884164ae24a053 received from {username='slave'} at 127.0.0.1:34930
I20240502 21:39:42.371115 4709 tablet_copy_service.cc:433] P 09ff1ee782864ac99d05a7f011aeef44: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-856dc0c130c04f4894884164ae24a053 on tablet 856dc0c130c04f4894884164ae24a053 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:42.401605 5409 raft_consensus.cc:1203] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:42.609576 5409 raft_consensus.cc:1203] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:42.759860 5409 raft_consensus.cc:1203] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:42.787442 5409 raft_consensus.cc:1203] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:42.836973 5441 raft_consensus.cc:1050] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:42.840121 5441 consensus_queue.cc:237] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:42.847779 5409 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:42.849505 4772 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:42.850536 5104 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:42.853129 4694 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:42.853399 5104 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:42.854398 5441 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:42.863132 5493 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.868999 4772 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.875435 4694 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.882206 5409 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.889504 4592 catalog_manager.cc:5484] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:42.928615 4851 consensus_queue.cc:237] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:42.944243 5409 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:42.946275 5441 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:42.950955 4771 raft_consensus.cc:1261] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:42.953330 5493 consensus_queue.cc:1035] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:42.956252 5441 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.967473 4772 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.976104 5409 raft_consensus.cc:2932] T 047156e0297249998c3f2c5362c70432 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:42.984934 4578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 047156e0297249998c3f2c5362c70432 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:42.996260 4594 catalog_manager.cc:5484] T 047156e0297249998c3f2c5362c70432 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.017910 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 047156e0297249998c3f2c5362c70432 with delete_type TABLET_DATA_TOMBSTONED (TS 09ff1ee782864ac99d05a7f011aeef44 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53982
I20240502 21:39:43.020453 5509 tablet_replica.cc:330] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: stopping tablet replica
I20240502 21:39:43.021733 5509 raft_consensus.cc:2229] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:43.024477 5509 raft_consensus.cc:2258] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:43.030625 5509 ts_tablet_manager.cc:1875] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:43.049481 5509 ts_tablet_manager.cc:1888] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:43.049945 5509 log.cc:1196] T 047156e0297249998c3f2c5362c70432 P 09ff1ee782864ac99d05a7f011aeef44: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/wals/047156e0297249998c3f2c5362c70432
I20240502 21:39:43.051620 4581 catalog_manager.cc:4830] TS 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): tablet 047156e0297249998c3f2c5362c70432 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:43.145253 5493 raft_consensus.cc:1050] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:43.147668 5493 consensus_queue.cc:237] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.153553 4771 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.154240 5409 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.154840 4698 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.155750 5490 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:43.157336 5441 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:43.161036 5493 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:43.173100 5493 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.182516 4772 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.186519 4698 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.195087 4594 catalog_manager.cc:5484] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:43.175208 5409 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.221223 4852 consensus_queue.cc:237] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.231458 4694 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.234493 5493 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:43.240242 5409 raft_consensus.cc:1261] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.244910 5441 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.256168 4698 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.258376 5190 raft_consensus.cc:1050] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:43.258256 5493 consensus_queue.cc:1035] T 1c65cfc531344398b70ce5862b2211fd P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:43.262574 5190 consensus_queue.cc:237] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.264824 5408 raft_consensus.cc:2932] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.270857 5407 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.273716 5459 raft_consensus.cc:1050] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:43.278903 4852 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.280680 4694 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.281975 5494 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:43.281785 5459 consensus_queue.cc:237] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.283288 4594 catalog_manager.cc:5484] T 1c65cfc531344398b70ce5862b2211fd P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.285025 5494 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:43.306795 5407 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.311491 4772 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.314042 5499 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:43.314736 4848 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:43.315820 4759 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c65cfc531344398b70ce5862b2211fd with delete_type TABLET_DATA_TOMBSTONED (TS ddea410051614f23b93c7342ec7cc545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39510
I20240502 21:39:43.316169 5499 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20240502 21:39:43.335635 5190 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:43.323942 5499 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240502 21:39:43.374209 5498 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.375687 4851 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.394393 4578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c65cfc531344398b70ce5862b2211fd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:43.395025 5518 tablet_replica.cc:330] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: stopping tablet replica
I20240502 21:39:43.396082 5518 raft_consensus.cc:2229] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:43.396842 5518 raft_consensus.cc:2258] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:43.400966 5198 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.402561 5518 ts_tablet_manager.cc:1875] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:43.404850 4698 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.411829 4851 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.418406 5408 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.425606 5407 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.431878 4594 catalog_manager.cc:5484] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:43.447247 4771 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.450249 5518 ts_tablet_manager.cc:1888] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:43.450839 5518 log.cc:1196] T 1c65cfc531344398b70ce5862b2211fd P ddea410051614f23b93c7342ec7cc545: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/wals/1c65cfc531344398b70ce5862b2211fd
I20240502 21:39:43.453152 4580 catalog_manager.cc:4830] TS ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): tablet 1c65cfc531344398b70ce5862b2211fd (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:43.459021 4592 catalog_manager.cc:5484] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:43.487138 4770 consensus_queue.cc:237] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.498270 5407 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.499370 4851 raft_consensus.cc:1261] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.499913 5190 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:43.502837 5494 consensus_queue.cc:1035] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:43.509315 4698 consensus_queue.cc:237] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:43.510488 5494 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.515324 4852 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.517032 5407 raft_consensus.cc:2932] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.528652 5408 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.529728 4770 raft_consensus.cc:1261] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:43.531019 4580 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80c35cadfff54236bbf05e32f945728f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:43.531502 5498 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:43.533629 5498 consensus_queue.cc:1035] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:43.537850 4593 catalog_manager.cc:5484] T 80c35cadfff54236bbf05e32f945728f P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.543916 5525 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.544730 4771 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.557850 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c35cadfff54236bbf05e32f945728f with delete_type TABLET_DATA_TOMBSTONED (TS 09ff1ee782864ac99d05a7f011aeef44 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53982
I20240502 21:39:43.587289 5407 raft_consensus.cc:2932] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.703105 5526 tablet_replica.cc:330] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: stopping tablet replica
I20240502 21:39:43.704105 5526 raft_consensus.cc:2229] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:43.705003 5526 raft_consensus.cc:2258] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:43.815246 4581 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 856dc0c130c04f4894884164ae24a053 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:43.831352 5526 ts_tablet_manager.cc:1875] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:43.858336 5526 ts_tablet_manager.cc:1888] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:43.858842 5526 log.cc:1196] T 80c35cadfff54236bbf05e32f945728f P 09ff1ee782864ac99d05a7f011aeef44: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/wals/80c35cadfff54236bbf05e32f945728f
I20240502 21:39:43.861718 4593 catalog_manager.cc:5484] T 856dc0c130c04f4894884164ae24a053 P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:43.885885 4581 catalog_manager.cc:4830] TS 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): tablet 80c35cadfff54236bbf05e32f945728f (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:43.895895 4834 tablet_service.cc:1514] Processing DeleteTablet for tablet 856dc0c130c04f4894884164ae24a053 with delete_type TABLET_DATA_TOMBSTONED (TS df8e5a877b6640d692917083c9fb5282 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42764
I20240502 21:39:43.915678 5530 tablet_replica.cc:330] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: stopping tablet replica
I20240502 21:39:43.928565 5530 raft_consensus.cc:2229] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:43.929317 5530 raft_consensus.cc:2258] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:43.932471 5530 ts_tablet_manager.cc:1875] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:43.944944 5530 ts_tablet_manager.cc:1888] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:43.945333 5530 log.cc:1196] T 856dc0c130c04f4894884164ae24a053 P df8e5a877b6640d692917083c9fb5282: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/wals/856dc0c130c04f4894884164ae24a053
I20240502 21:39:43.946785 4578 catalog_manager.cc:4830] TS df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): tablet 856dc0c130c04f4894884164ae24a053 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
W20240502 21:39:44.648929 4626 debug-util.cc:398] Leaking SignalData structure 0x7b080003f460 after lost signal to thread 3530
W20240502 21:39:44.649870 4626 debug-util.cc:398] Leaking SignalData structure 0x7b080003fb80 after lost signal to thread 4629
W20240502 21:39:44.650909 4626 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6b00 after lost signal to thread 4723
W20240502 21:39:44.651700 4626 debug-util.cc:398] Leaking SignalData structure 0x7b0800312f60 after lost signal to thread 4801
W20240502 21:39:44.652500 4626 debug-util.cc:398] Leaking SignalData structure 0x7b0800283e60 after lost signal to thread 4876
W20240502 21:39:44.653295 4626 debug-util.cc:398] Leaking SignalData structure 0x7b0800172720 after lost signal to thread 5433
I20240502 21:39:45.003960 4850 consensus_queue.cc:237] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.010907 4699 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.011057 4771 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.012436 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:45.013387 5531 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:39:45.015435 4816 consensus_peers.cc:485] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96d93565463749aab6550e43871f7730. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:45.023814 5532 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.025796 4771 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.047402 4698 consensus_queue.cc:237] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.051992 4594 catalog_manager.cc:5484] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.041560 4699 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.081449 4850 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.083328 5533 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 21:39:45.087047 4663 consensus_peers.cc:485] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302a987ffb0842c0bfae54675c665763. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:45.089845 4771 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.092016 5498 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:45.143779 5525 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.145920 4850 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.155210 4771 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.162246 4774 consensus_queue.cc:237] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
W20240502 21:39:45.182220 4738 consensus_peers.cc:485] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df2394febdff4bb8bd495e4b8086e412. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:45.185477 4850 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.187784 5528 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:45.178670 4591 catalog_manager.cc:5484] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.194167 4698 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.198899 5528 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:45.202819 5546 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.213716 4850 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.221052 4695 consensus_queue.cc:237] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.241552 4698 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.252667 4848 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:39:45.252774 4663 consensus_peers.cc:485] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f14af5e6ab984917b771589cdb45a779. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:45.255054 4774 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:45.258407 5495 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:45.259658 5498 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:45.237762 4593 catalog_manager.cc:5484] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.290700 5525 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.294317 4774 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.319662 4850 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.337435 4593 catalog_manager.cc:5484] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:45.462208 5551 ts_tablet_manager.cc:902] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:45.472198 5552 ts_tablet_manager.cc:902] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:45.511404 5551 tablet_copy_client.cc:308] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.193:44335
I20240502 21:39:45.516099 4709 tablet_copy_service.cc:140] P 09ff1ee782864ac99d05a7f011aeef44: Received BeginTabletCopySession request for tablet 302a987ffb0842c0bfae54675c665763 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:34930)
I20240502 21:39:45.516713 4709 tablet_copy_service.cc:161] P 09ff1ee782864ac99d05a7f011aeef44: Beginning new tablet copy session on tablet 302a987ffb0842c0bfae54675c665763 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:34930: session id = 6a9bb5562dce4197b3ca1f398c12954c-302a987ffb0842c0bfae54675c665763
I20240502 21:39:45.523326 4709 tablet_copy_source_session.cc:205] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:45.529340 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302a987ffb0842c0bfae54675c665763. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:45.545325 5552 tablet_copy_client.cc:308] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.195:45259
I20240502 21:39:45.549289 4862 tablet_copy_service.cc:140] P df8e5a877b6640d692917083c9fb5282: Received BeginTabletCopySession request for tablet 96d93565463749aab6550e43871f7730 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:47828)
I20240502 21:39:45.549909 4862 tablet_copy_service.cc:161] P df8e5a877b6640d692917083c9fb5282: Beginning new tablet copy session on tablet 96d93565463749aab6550e43871f7730 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:47828: session id = 6a9bb5562dce4197b3ca1f398c12954c-96d93565463749aab6550e43871f7730
I20240502 21:39:45.557878 4862 tablet_copy_source_session.cc:205] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:45.561517 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d93565463749aab6550e43871f7730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:45.561873 5551 tablet_copy_client.cc:783] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:45.565006 5551 tablet_copy_client.cc:647] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:45.583778 5552 tablet_copy_client.cc:783] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:45.584610 5552 tablet_copy_client.cc:647] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:45.592020 5551 tablet_copy_client.cc:523] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:45.601871 5552 tablet_copy_client.cc:523] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:45.611125 5551 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:45.614606 5552 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:45.649322 5551 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:45.650386 5551 tablet_bootstrap.cc:492] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:45.651208 5551 ts_tablet_manager.cc:1367] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.006s
I20240502 21:39:45.654103 5551 raft_consensus.cc:348] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.655179 5551 raft_consensus.cc:729] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:45.655967 5551 consensus_queue.cc:260] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.686189 5551 ts_tablet_manager.cc:1398] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.035s user 0.013s sys 0.014s
I20240502 21:39:45.691200 4709 tablet_copy_service.cc:342] P 09ff1ee782864ac99d05a7f011aeef44: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-302a987ffb0842c0bfae54675c665763 received from {username='slave'} at 127.0.0.1:34930
I20240502 21:39:45.691656 4709 tablet_copy_service.cc:433] P 09ff1ee782864ac99d05a7f011aeef44: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-302a987ffb0842c0bfae54675c665763 on tablet 302a987ffb0842c0bfae54675c665763 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:45.704176 5552 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:45.705540 5552 tablet_bootstrap.cc:492] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:45.706406 5552 ts_tablet_manager.cc:1367] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.092s user 0.027s sys 0.019s
I20240502 21:39:45.709151 5552 raft_consensus.cc:348] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.716528 5552 raft_consensus.cc:729] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:45.717280 5552 consensus_queue.cc:260] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.715315 5551 ts_tablet_manager.cc:902] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:45.723199 5552 ts_tablet_manager.cc:1398] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240502 21:39:45.728603 4862 tablet_copy_service.cc:342] P df8e5a877b6640d692917083c9fb5282: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-96d93565463749aab6550e43871f7730 received from {username='slave'} at 127.0.0.1:47828
I20240502 21:39:45.729070 4862 tablet_copy_service.cc:433] P df8e5a877b6640d692917083c9fb5282: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-96d93565463749aab6550e43871f7730 on tablet 96d93565463749aab6550e43871f7730 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:45.742630 5552 ts_tablet_manager.cc:902] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:45.776364 5551 tablet_copy_client.cc:308] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.193:44335
I20240502 21:39:45.781364 4709 tablet_copy_service.cc:140] P 09ff1ee782864ac99d05a7f011aeef44: Received BeginTabletCopySession request for tablet f14af5e6ab984917b771589cdb45a779 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:34930)
I20240502 21:39:45.781945 4709 tablet_copy_service.cc:161] P 09ff1ee782864ac99d05a7f011aeef44: Beginning new tablet copy session on tablet f14af5e6ab984917b771589cdb45a779 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:34930: session id = 6a9bb5562dce4197b3ca1f398c12954c-f14af5e6ab984917b771589cdb45a779
I20240502 21:39:45.791461 4709 tablet_copy_source_session.cc:205] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:45.798362 5552 tablet_copy_client.cc:308] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.194:38679
I20240502 21:39:45.801509 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14af5e6ab984917b771589cdb45a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:45.801682 4786 tablet_copy_service.cc:140] P ddea410051614f23b93c7342ec7cc545: Received BeginTabletCopySession request for tablet df2394febdff4bb8bd495e4b8086e412 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:55718)
I20240502 21:39:45.804162 4786 tablet_copy_service.cc:161] P ddea410051614f23b93c7342ec7cc545: Beginning new tablet copy session on tablet df2394febdff4bb8bd495e4b8086e412 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:55718: session id = 6a9bb5562dce4197b3ca1f398c12954c-df2394febdff4bb8bd495e4b8086e412
I20240502 21:39:45.823113 5551 tablet_copy_client.cc:783] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:45.823851 5551 tablet_copy_client.cc:647] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:45.830125 4786 tablet_copy_source_session.cc:205] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:45.831480 5551 tablet_copy_client.cc:523] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:45.834084 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2394febdff4bb8bd495e4b8086e412. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:45.842645 5551 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:45.902501 5551 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:45.904225 5551 tablet_bootstrap.cc:492] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:45.905084 5551 ts_tablet_manager.cc:1367] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.064s user 0.031s sys 0.019s
I20240502 21:39:45.908088 5551 raft_consensus.cc:348] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.909220 5551 raft_consensus.cc:729] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:45.909881 5552 tablet_copy_client.cc:783] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:45.910701 5552 tablet_copy_client.cc:647] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:45.909955 5551 consensus_queue.cc:260] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.913370 5551 ts_tablet_manager.cc:1398] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240502 21:39:45.915246 4709 tablet_copy_service.cc:342] P 09ff1ee782864ac99d05a7f011aeef44: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-f14af5e6ab984917b771589cdb45a779 received from {username='slave'} at 127.0.0.1:34930
I20240502 21:39:45.915655 4709 tablet_copy_service.cc:433] P 09ff1ee782864ac99d05a7f011aeef44: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-f14af5e6ab984917b771589cdb45a779 on tablet f14af5e6ab984917b771589cdb45a779 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:45.919821 5552 tablet_copy_client.cc:523] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:45.936354 5552 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:45.977030 5552 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:45.978116 5552 tablet_bootstrap.cc:492] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:45.978904 5552 ts_tablet_manager.cc:1367] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.001s
I20240502 21:39:45.981817 5552 raft_consensus.cc:348] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.982817 5552 raft_consensus.cc:729] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:45.983561 5552 consensus_queue.cc:260] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:45.991978 5408 raft_consensus.cc:1203] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:45.989101 5552 ts_tablet_manager.cc:1398] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240502 21:39:46.003028 4786 tablet_copy_service.cc:342] P ddea410051614f23b93c7342ec7cc545: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-df2394febdff4bb8bd495e4b8086e412 received from {username='slave'} at 127.0.0.1:55718
I20240502 21:39:46.003553 4786 tablet_copy_service.cc:433] P ddea410051614f23b93c7342ec7cc545: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-df2394febdff4bb8bd495e4b8086e412 on tablet df2394febdff4bb8bd495e4b8086e412 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:46.008833 5408 raft_consensus.cc:1203] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:46.158711 5408 raft_consensus.cc:1203] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:46.340358 5408 raft_consensus.cc:1203] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:46.379359 5531 raft_consensus.cc:1050] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:46.382416 5531 consensus_queue.cc:237] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.393900 4770 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.395746 5408 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.397177 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.410112 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.418565 4699 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.428078 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240502 21:39:46.487083 4817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 65685.8us
W20240502 21:39:46.494326 4818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 73665.6us
I20240502 21:39:46.496600 5561 raft_consensus.cc:1050] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:46.499121 5561 consensus_queue.cc:237] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.529336 5532 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.544345 5407 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.548990 5408 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.548086 4698 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.552660 4850 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.552527 5586 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:46.553712 4774 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.554476 5498 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.556114 5498 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.563694 5525 raft_consensus.cc:1050] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:46.576182 5525 consensus_queue.cc:237] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.579595 4771 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.592078 4591 catalog_manager.cc:5484] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:46.614859 5407 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.616708 5591 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:46.617563 4771 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.621618 5525 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.622862 4851 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.633232 5586 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.638430 4774 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.644829 5525 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.651567 5498 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.661204 4774 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.676033 5408 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.688980 4850 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.697105 5407 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.708667 4848 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.726933 4592 catalog_manager.cc:5484] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.728798 4852 consensus_queue.cc:237] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.730985 4592 catalog_manager.cc:5484] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.757076 4696 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:46.757843 5409 raft_consensus.cc:1261] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:46.758519 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:46.760639 5532 consensus_queue.cc:1035] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 21:39:46.771066 5583 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.782526 5408 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.778550 4696 raft_consensus.cc:2932] T 96d93565463749aab6550e43871f7730 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.794790 4578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96d93565463749aab6550e43871f7730 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:46.814904 4698 consensus_queue.cc:237] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.822510 5408 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:46.824187 5585 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:46.827530 4772 raft_consensus.cc:1261] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:46.830968 4593 catalog_manager.cc:5484] T 96d93565463749aab6550e43871f7730 P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.836201 5498 consensus_queue.cc:1035] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:46.840955 5408 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.843518 5561 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.844578 4774 raft_consensus.cc:2932] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.864337 4581 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 302a987ffb0842c0bfae54675c665763 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:46.881763 4591 catalog_manager.cc:5484] T 302a987ffb0842c0bfae54675c665763 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.906235 5577 raft_consensus.cc:1050] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:46.907738 4759 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d93565463749aab6550e43871f7730 with delete_type TABLET_DATA_TOMBSTONED (TS ddea410051614f23b93c7342ec7cc545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39510
I20240502 21:39:46.908610 4834 tablet_service.cc:1514] Processing DeleteTablet for tablet 302a987ffb0842c0bfae54675c665763 with delete_type TABLET_DATA_TOMBSTONED (TS df8e5a877b6640d692917083c9fb5282 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42764
I20240502 21:39:46.908818 5577 consensus_queue.cc:237] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:46.922580 5408 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.924291 5575 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.928242 4850 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.930066 5575 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:46.931699 4698 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:46.935372 5576 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:46.958676 5577 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.960834 4848 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.978302 5606 tablet_replica.cc:330] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: stopping tablet replica
I20240502 21:39:46.981339 5606 raft_consensus.cc:2229] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:46.982506 5606 raft_consensus.cc:2258] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:46.980418 4696 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.983366 4591 catalog_manager.cc:5484] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.986886 5606 ts_tablet_manager.cc:1875] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:46.979892 5604 tablet_replica.cc:330] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: stopping tablet replica
I20240502 21:39:46.988077 5408 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:46.995824 5604 raft_consensus.cc:2229] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:47.019235 4771 consensus_queue.cc:237] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:47.023298 5604 raft_consensus.cc:2258] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:47.031880 5606 ts_tablet_manager.cc:1888] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:47.032557 5606 log.cc:1196] T 96d93565463749aab6550e43871f7730 P ddea410051614f23b93c7342ec7cc545: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/wals/96d93565463749aab6550e43871f7730
I20240502 21:39:47.034461 4580 catalog_manager.cc:4830] TS ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): tablet 96d93565463749aab6550e43871f7730 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:47.035375 5604 ts_tablet_manager.cc:1875] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:47.072907 4850 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:47.073091 5408 raft_consensus.cc:1261] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:47.083195 5575 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:47.085489 5577 consensus_queue.cc:1035] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:47.096905 5576 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.103425 5408 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.109606 5604 ts_tablet_manager.cc:1888] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:47.110205 5604 log.cc:1196] T 302a987ffb0842c0bfae54675c665763 P df8e5a877b6640d692917083c9fb5282: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/wals/302a987ffb0842c0bfae54675c665763
I20240502 21:39:47.111904 4578 catalog_manager.cc:4830] TS df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): tablet 302a987ffb0842c0bfae54675c665763 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:47.119894 4580 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df2394febdff4bb8bd495e4b8086e412 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:47.129992 4591 catalog_manager.cc:5484] T df2394febdff4bb8bd495e4b8086e412 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 21:39:47.134285 4817 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 52064.9us
I20240502 21:39:47.151852 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet df2394febdff4bb8bd495e4b8086e412 with delete_type TABLET_DATA_TOMBSTONED (TS 09ff1ee782864ac99d05a7f011aeef44 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53982
I20240502 21:39:47.168717 4848 raft_consensus.cc:2932] T df2394febdff4bb8bd495e4b8086e412 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.169906 5611 tablet_replica.cc:330] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: stopping tablet replica
I20240502 21:39:47.213249 5611 raft_consensus.cc:2229] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:47.214064 5611 raft_consensus.cc:2258] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:47.217878 5611 ts_tablet_manager.cc:1875] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:47.233650 5611 ts_tablet_manager.cc:1888] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:47.234191 5611 log.cc:1196] T df2394febdff4bb8bd495e4b8086e412 P 09ff1ee782864ac99d05a7f011aeef44: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/wals/df2394febdff4bb8bd495e4b8086e412
I20240502 21:39:47.235807 4581 catalog_manager.cc:4830] TS 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): tablet df2394febdff4bb8bd495e4b8086e412 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:47.250804 4698 consensus_queue.cc:237] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:47.261574 5408 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:47.263218 5525 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:47.267951 4848 raft_consensus.cc:1261] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:47.274471 5498 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.290719 5525 consensus_queue.cc:1035] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:47.286048 5408 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.292927 4581 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f14af5e6ab984917b771589cdb45a779 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:47.298075 4847 raft_consensus.cc:2932] T f14af5e6ab984917b771589cdb45a779 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:47.311030 4591 catalog_manager.cc:5484] T f14af5e6ab984917b771589cdb45a779 P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:47.328799 4759 tablet_service.cc:1514] Processing DeleteTablet for tablet f14af5e6ab984917b771589cdb45a779 with delete_type TABLET_DATA_TOMBSTONED (TS ddea410051614f23b93c7342ec7cc545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39510
I20240502 21:39:47.329663 5606 tablet_replica.cc:330] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: stopping tablet replica
I20240502 21:39:47.330653 5606 raft_consensus.cc:2229] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:47.331359 5606 raft_consensus.cc:2258] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:47.334676 5606 ts_tablet_manager.cc:1875] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:47.351783 5606 ts_tablet_manager.cc:1888] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:47.352244 5606 log.cc:1196] T f14af5e6ab984917b771589cdb45a779 P ddea410051614f23b93c7342ec7cc545: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/wals/f14af5e6ab984917b771589cdb45a779
I20240502 21:39:47.354033 4580 catalog_manager.cc:4830] TS ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): tablet f14af5e6ab984917b771589cdb45a779 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
W20240502 21:39:48.226442 4737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 49923.7us
I20240502 21:39:48.398061 4770 consensus_queue.cc:237] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:48.405385 4847 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:39:48.405464 4738 consensus_peers.cc:485] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6171c5f680040b9a3efc1f98c456ea9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:48.408453 5610 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:48.411160 4698 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.412429 5610 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:48.459103 5610 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.462384 4848 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.479005 4848 consensus_queue.cc:237] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:48.480180 4591 catalog_manager.cc:5484] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.485585 4698 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.504467 4696 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.504657 4770 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.507329 5614 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:48.516814 5613 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240502 21:39:48.521880 4816 consensus_peers.cc:485] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 238a1f48ddb14387bc2aa4419ff8879e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:48.533330 5582 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.551986 4770 consensus_queue.cc:237] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:48.554620 4771 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.556797 4696 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:48.585191 4738 consensus_peers.cc:485] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd1bcf32e7444ed80e0f7ae05786dde. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:48.587302 4848 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.596740 5621 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:48.599125 4698 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.607738 4594 catalog_manager.cc:5484] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:48.616482 5610 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:48.634316 5575 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.645516 4696 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.653592 4698 consensus_queue.cc:237] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:48.656829 4594 catalog_manager.cc:5484] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:48.670045 4847 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.671833 5617 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:48.672358 4848 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:48.679152 4663 consensus_peers.cc:485] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1a911c9636f426e9d08bee71f60eaab. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:48.679869 4770 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:48.682246 5619 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:48.712293 5619 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.718114 4847 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:48.742470 4591 catalog_manager.cc:5484] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:48.782635 4774 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
W20240502 21:39:48.794430 5635 log.cc:924] Time spent T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Append to log took a long time: real 0.059s user 0.004s sys 0.000s
W20240502 21:39:48.804519 4737 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.195:45259, user_credentials={real_user=slave}} blocked reactor thread for 43972.5us
I20240502 21:39:48.864831 5638 ts_tablet_manager.cc:902] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:48.890616 5638 tablet_copy_client.cc:308] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.194:38679
I20240502 21:39:48.892444 4786 tablet_copy_service.cc:140] P ddea410051614f23b93c7342ec7cc545: Received BeginTabletCopySession request for tablet e6171c5f680040b9a3efc1f98c456ea9 from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:55718)
I20240502 21:39:48.893018 4786 tablet_copy_service.cc:161] P ddea410051614f23b93c7342ec7cc545: Beginning new tablet copy session on tablet e6171c5f680040b9a3efc1f98c456ea9 from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:55718: session id = 6a9bb5562dce4197b3ca1f398c12954c-e6171c5f680040b9a3efc1f98c456ea9
I20240502 21:39:48.901420 4786 tablet_copy_source_session.cc:205] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:48.907287 5638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6171c5f680040b9a3efc1f98c456ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:48.909185 5643 ts_tablet_manager.cc:902] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
I20240502 21:39:48.926359 5638 tablet_copy_client.cc:783] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:48.927497 5638 tablet_copy_client.cc:647] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:48.935914 5638 tablet_copy_client.cc:523] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:48.948900 5638 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:48.978379 5643 tablet_copy_client.cc:308] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.195:45259
I20240502 21:39:48.982156 4862 tablet_copy_service.cc:140] P df8e5a877b6640d692917083c9fb5282: Received BeginTabletCopySession request for tablet 238a1f48ddb14387bc2aa4419ff8879e from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:47828)
I20240502 21:39:48.982795 4862 tablet_copy_service.cc:161] P df8e5a877b6640d692917083c9fb5282: Beginning new tablet copy session on tablet 238a1f48ddb14387bc2aa4419ff8879e from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:47828: session id = 6a9bb5562dce4197b3ca1f398c12954c-238a1f48ddb14387bc2aa4419ff8879e
I20240502 21:39:48.986299 5638 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:48.987243 5638 tablet_bootstrap.cc:492] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:48.994782 5638 ts_tablet_manager.cc:1367] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.013s
I20240502 21:39:48.996219 4862 tablet_copy_source_session.cc:205] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:48.999575 5638 raft_consensus.cc:348] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.004228 5638 raft_consensus.cc:729] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:49.005134 5638 consensus_queue.cc:260] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.005896 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238a1f48ddb14387bc2aa4419ff8879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:49.014490 5638 ts_tablet_manager.cc:1398] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.019s user 0.008s sys 0.004s
I20240502 21:39:49.016389 4786 tablet_copy_service.cc:342] P ddea410051614f23b93c7342ec7cc545: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-e6171c5f680040b9a3efc1f98c456ea9 received from {username='slave'} at 127.0.0.1:55718
I20240502 21:39:49.016878 4786 tablet_copy_service.cc:433] P ddea410051614f23b93c7342ec7cc545: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-e6171c5f680040b9a3efc1f98c456ea9 on tablet e6171c5f680040b9a3efc1f98c456ea9 with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:49.031128 5643 tablet_copy_client.cc:783] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:49.032229 5643 tablet_copy_client.cc:647] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:49.038894 5643 tablet_copy_client.cc:523] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:49.048744 5643 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:49.070376 5643 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:49.072724 5643 tablet_bootstrap.cc:492] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:49.075117 5643 ts_tablet_manager.cc:1367] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.005s
I20240502 21:39:49.078141 5643 raft_consensus.cc:348] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.079231 5643 raft_consensus.cc:729] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:49.080041 5643 consensus_queue.cc:260] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.083003 5643 ts_tablet_manager.cc:1398] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:39:49.084285 5638 ts_tablet_manager.cc:902] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
I20240502 21:39:49.091814 4862 tablet_copy_service.cc:342] P df8e5a877b6640d692917083c9fb5282: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-238a1f48ddb14387bc2aa4419ff8879e received from {username='slave'} at 127.0.0.1:47828
I20240502 21:39:49.092283 4862 tablet_copy_service.cc:433] P df8e5a877b6640d692917083c9fb5282: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-238a1f48ddb14387bc2aa4419ff8879e on tablet 238a1f48ddb14387bc2aa4419ff8879e with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:49.143473 5638 tablet_copy_client.cc:308] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.193:44335
I20240502 21:39:49.145903 4709 tablet_copy_service.cc:140] P 09ff1ee782864ac99d05a7f011aeef44: Received BeginTabletCopySession request for tablet a1a911c9636f426e9d08bee71f60eaab from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:34930)
I20240502 21:39:49.146489 4709 tablet_copy_service.cc:161] P 09ff1ee782864ac99d05a7f011aeef44: Beginning new tablet copy session on tablet a1a911c9636f426e9d08bee71f60eaab from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:34930: session id = 6a9bb5562dce4197b3ca1f398c12954c-a1a911c9636f426e9d08bee71f60eaab
I20240502 21:39:49.155759 4709 tablet_copy_source_session.cc:205] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:49.161319 5638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a911c9636f426e9d08bee71f60eaab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:49.182626 5638 tablet_copy_client.cc:783] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:49.183409 5638 tablet_copy_client.cc:647] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:49.188369 5638 tablet_copy_client.cc:523] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:49.191890 5643 ts_tablet_manager.cc:902] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
I20240502 21:39:49.201722 5638 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:49.250516 5638 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:49.251974 5638 tablet_bootstrap.cc:492] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:49.252820 5638 ts_tablet_manager.cc:1367] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.008s
I20240502 21:39:49.256155 5638 raft_consensus.cc:348] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.257418 5638 raft_consensus.cc:729] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:49.258172 5638 consensus_queue.cc:260] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.261417 5638 ts_tablet_manager.cc:1398] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240502 21:39:49.284963 4709 tablet_copy_service.cc:342] P 09ff1ee782864ac99d05a7f011aeef44: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-a1a911c9636f426e9d08bee71f60eaab received from {username='slave'} at 127.0.0.1:34930
I20240502 21:39:49.285483 4709 tablet_copy_service.cc:433] P 09ff1ee782864ac99d05a7f011aeef44: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-a1a911c9636f426e9d08bee71f60eaab on tablet a1a911c9636f426e9d08bee71f60eaab with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:49.323719 5408 raft_consensus.cc:1203] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:49.393977 5643 tablet_copy_client.cc:308] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.113.194:38679
I20240502 21:39:49.401330 5407 raft_consensus.cc:1203] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:49.402765 4786 tablet_copy_service.cc:140] P ddea410051614f23b93c7342ec7cc545: Received BeginTabletCopySession request for tablet 1cd1bcf32e7444ed80e0f7ae05786dde from peer 6a9bb5562dce4197b3ca1f398c12954c ({username='slave'} at 127.0.0.1:55718)
I20240502 21:39:49.403350 4786 tablet_copy_service.cc:161] P ddea410051614f23b93c7342ec7cc545: Beginning new tablet copy session on tablet 1cd1bcf32e7444ed80e0f7ae05786dde from peer 6a9bb5562dce4197b3ca1f398c12954c at {username='slave'} at 127.0.0.1:55718: session id = 6a9bb5562dce4197b3ca1f398c12954c-1cd1bcf32e7444ed80e0f7ae05786dde
I20240502 21:39:49.422155 4786 tablet_copy_source_session.cc:205] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:39:49.425585 5643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd1bcf32e7444ed80e0f7ae05786dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:39:49.443120 5643 tablet_copy_client.cc:783] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 0 data blocks...
I20240502 21:39:49.443749 5643 tablet_copy_client.cc:647] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:39:49.451289 5643 tablet_copy_client.cc:523] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:39:49.462919 5643 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap starting.
I20240502 21:39:49.488948 5643 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:39:49.490063 5643 tablet_bootstrap.cc:492] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Bootstrap complete.
I20240502 21:39:49.490867 5643 ts_tablet_manager.cc:1367] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.005s
I20240502 21:39:49.494031 5643 raft_consensus.cc:348] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.495136 5643 raft_consensus.cc:729] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a9bb5562dce4197b3ca1f398c12954c, State: Initialized, Role: LEARNER
I20240502 21:39:49.495968 5643 consensus_queue.cc:260] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:49.499212 5643 ts_tablet_manager.cc:1398] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:39:49.501271 4786 tablet_copy_service.cc:342] P ddea410051614f23b93c7342ec7cc545: Request end of tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-1cd1bcf32e7444ed80e0f7ae05786dde received from {username='slave'} at 127.0.0.1:55718
I20240502 21:39:49.501778 4786 tablet_copy_service.cc:433] P ddea410051614f23b93c7342ec7cc545: ending tablet copy session 6a9bb5562dce4197b3ca1f398c12954c-1cd1bcf32e7444ed80e0f7ae05786dde on tablet 1cd1bcf32e7444ed80e0f7ae05786dde with peer 6a9bb5562dce4197b3ca1f398c12954c
I20240502 21:39:49.537920 5407 raft_consensus.cc:1203] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:49.805810 5637 raft_consensus.cc:1050] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:49.810250 5407 raft_consensus.cc:1203] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:39:49.819634 5637 consensus_queue.cc:237] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:49.832680 5407 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.833189 4847 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.834215 4698 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.834589 5637 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:49.835727 5610 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:39:49.838927 5662 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:49.856062 5637 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:49.863056 5407 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:49.881798 4847 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:49.904917 4696 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:49.922806 5613 raft_consensus.cc:1050] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:49.925576 5613 consensus_queue.cc:237] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:49.934427 4770 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.934656 5407 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.936120 5670 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:49.969398 4591 catalog_manager.cc:5484] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:49.976110 4696 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:49.991971 5670 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.065064 5670 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.100631 5617 raft_consensus.cc:1050] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:50.114751 5617 consensus_queue.cc:237] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:50.185911 5672 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.189422 5407 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
W20240502 21:39:50.209635 4664 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.194:38679, user_credentials={real_user=slave}} blocked reactor thread for 66714.9us
I20240502 21:39:50.256197 4774 consensus_queue.cc:237] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:50.259385 5407 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.251026 4848 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.269927 5679 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.248019 4590 catalog_manager.cc:5484] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.272271 5617 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.248700 4772 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.293078 5407 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.296499 5637 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:50.300987 5662 raft_consensus.cc:1050] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545: attempting to promote NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c to VOTER
I20240502 21:39:50.307581 4696 raft_consensus.cc:1261] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.306638 5668 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.311403 5408 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.304112 5662 consensus_queue.cc:237] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:50.333981 4590 catalog_manager.cc:5484] T e6171c5f680040b9a3efc1f98c456ea9 P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.340085 4580 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6171c5f680040b9a3efc1f98c456ea9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:50.345639 5680 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.350093 4699 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.362735 4848 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.367776 4847 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.370826 5637 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.374953 4834 tablet_service.cc:1514] Processing DeleteTablet for tablet e6171c5f680040b9a3efc1f98c456ea9 with delete_type TABLET_DATA_TOMBSTONED (TS df8e5a877b6640d692917083c9fb5282 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42764
I20240502 21:39:50.378279 5409 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 LEARNER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.379417 5408 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.385644 5662 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.395988 4772 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.401325 5610 consensus_queue.cc:1035] T e6171c5f680040b9a3efc1f98c456ea9 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:50.420620 4593 catalog_manager.cc:5484] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:50.442023 4849 consensus_queue.cc:237] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:50.454190 5407 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.456442 5686 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:39:50.463732 4695 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:39:50.471431 5690 tablet_replica.cc:330] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: stopping tablet replica
I20240502 21:39:50.478688 4699 raft_consensus.cc:1261] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.504634 5408 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.499991 5613 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
W20240502 21:39:50.516660 4740 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.113.193:44335, user_credentials={real_user=slave}} blocked reactor thread for 109933us
I20240502 21:39:50.519316 5662 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 21:39:50.498330 5670 consensus_queue.cc:1035] T 238a1f48ddb14387bc2aa4419ff8879e P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20240502 21:39:50.523521 5677 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240502 21:39:50.542670 4699 raft_consensus.cc:2932] T e6171c5f680040b9a3efc1f98c456ea9 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.556190 4578 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 238a1f48ddb14387bc2aa4419ff8879e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:50.556835 5690 raft_consensus.cc:2229] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:50.557840 5690 raft_consensus.cc:2258] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:50.559669 5692 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.562275 5409 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.570176 5690 ts_tablet_manager.cc:1875] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:50.579843 4594 catalog_manager.cc:5484] T 238a1f48ddb14387bc2aa4419ff8879e P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.584095 4698 raft_consensus.cc:2932] T 238a1f48ddb14387bc2aa4419ff8879e P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ddea410051614f23b93c7342ec7cc545 (127.3.113.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.602180 4759 tablet_service.cc:1514] Processing DeleteTablet for tablet 238a1f48ddb14387bc2aa4419ff8879e with delete_type TABLET_DATA_TOMBSTONED (TS ddea410051614f23b93c7342ec7cc545 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39510
I20240502 21:39:50.614192 4771 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.630582 4698 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.641513 5696 tablet_replica.cc:330] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: stopping tablet replica
I20240502 21:39:50.642619 5696 raft_consensus.cc:2229] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:50.653728 4847 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.654891 4594 catalog_manager.cc:5484] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index 2 to 3, 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:39:50.661943 5696 raft_consensus.cc:2258] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:50.666324 5696 ts_tablet_manager.cc:1875] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:50.669203 4771 consensus_queue.cc:237] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:50.679607 5409 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.681310 5692 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:50.684538 5690 ts_tablet_manager.cc:1888] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:50.687047 5690 log.cc:1196] T e6171c5f680040b9a3efc1f98c456ea9 P df8e5a877b6640d692917083c9fb5282: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/wals/e6171c5f680040b9a3efc1f98c456ea9
I20240502 21:39:50.689747 4578 catalog_manager.cc:4830] TS df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): tablet e6171c5f680040b9a3efc1f98c456ea9 (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:50.692067 5662 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.697932 4852 raft_consensus.cc:1261] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:50.700467 5408 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.701814 5610 consensus_queue.cc:1035] T 1cd1bcf32e7444ed80e0f7ae05786dde P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:50.705559 4847 raft_consensus.cc:2932] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.724539 4580 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1cd1bcf32e7444ed80e0f7ae05786dde with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:50.725292 4590 catalog_manager.cc:5484] T 1cd1bcf32e7444ed80e0f7ae05786dde P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index 3 to 4, VOTER 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193) evicted. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:50.736932 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd1bcf32e7444ed80e0f7ae05786dde with delete_type TABLET_DATA_TOMBSTONED (TS 09ff1ee782864ac99d05a7f011aeef44 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53982
I20240502 21:39:50.739562 5697 tablet_replica.cc:330] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: stopping tablet replica
I20240502 21:39:50.740844 5697 raft_consensus.cc:2229] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:50.741672 5697 raft_consensus.cc:2258] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:50.760344 5696 ts_tablet_manager.cc:1888] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:50.767459 5696 log.cc:1196] T 238a1f48ddb14387bc2aa4419ff8879e P ddea410051614f23b93c7342ec7cc545: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-1-root/wals/238a1f48ddb14387bc2aa4419ff8879e
I20240502 21:39:50.768107 5697 ts_tablet_manager.cc:1875] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:50.771519 4580 catalog_manager.cc:4830] TS ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679): tablet 238a1f48ddb14387bc2aa4419ff8879e (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:50.794106 5697 ts_tablet_manager.cc:1888] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:50.794732 5697 log.cc:1196] T 1cd1bcf32e7444ed80e0f7ae05786dde P 09ff1ee782864ac99d05a7f011aeef44: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-0-root/wals/1cd1bcf32e7444ed80e0f7ae05786dde
I20240502 21:39:50.797156 4581 catalog_manager.cc:4830] TS 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335): tablet 1cd1bcf32e7444ed80e0f7ae05786dde (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:51.072968 4699 consensus_queue.cc:237] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } }
I20240502 21:39:51.088495 5408 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:51.090114 5679 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:51.090446 4772 raft_consensus.cc:1261] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:39:51.094448 5676 consensus_queue.cc:1035] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:39:51.099033 5676 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:51.100134 5408 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:51.118868 4772 raft_consensus.cc:2932] T a1a911c9636f426e9d08bee71f60eaab P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:51.118868 4592 catalog_manager.cc:5484] T a1a911c9636f426e9d08bee71f60eaab P 6a9bb5562dce4197b3ca1f398c12954c reported cstate change: config changed from index 3 to 4, VOTER df8e5a877b6640d692917083c9fb5282 (127.3.113.195) evicted. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: false } } }
I20240502 21:39:51.131018 4581 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1a911c9636f426e9d08bee71f60eaab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:39:51.177168 4834 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a911c9636f426e9d08bee71f60eaab with delete_type TABLET_DATA_TOMBSTONED (TS df8e5a877b6640d692917083c9fb5282 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42764
I20240502 21:39:51.178128 5690 tablet_replica.cc:330] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: stopping tablet replica
I20240502 21:39:51.179186 5690 raft_consensus.cc:2229] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:39:51.179849 5690 raft_consensus.cc:2258] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:39:51.182745 5690 ts_tablet_manager.cc:1875] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:39:51.196062 5690 ts_tablet_manager.cc:1888] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:39:51.196514 5690 log.cc:1196] T a1a911c9636f426e9d08bee71f60eaab P df8e5a877b6640d692917083c9fb5282: Deleting WAL directory at /tmp/dist-test-taskbxjmaC/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714685909228215-3527-0/minicluster-data/ts-2-root/wals/a1a911c9636f426e9d08bee71f60eaab
I20240502 21:39:51.197928 4578 catalog_manager.cc:4830] TS df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259): tablet a1a911c9636f426e9d08bee71f60eaab (table test-workload [id=b776127a918741d98edf5503b38d35bf]) successfully deleted
I20240502 21:39:52.223361 4699 consensus_queue.cc:237] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:52.229815 4847 raft_consensus.cc:1261] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:39:52.230089 4663 consensus_peers.cc:485] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 515491e619e145a292c5e5f654cc63ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:52.231925 4771 raft_consensus.cc:1261] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer 09ff1ee782864ac99d05a7f011aeef44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.231788 5699 consensus_queue.cc:1035] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:52.233067 5698 consensus_queue.cc:1035] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:52.243330 5698 raft_consensus.cc:2932] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.245697 4771 raft_consensus.cc:2932] T 515491e619e145a292c5e5f654cc63ee P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.245321 4847 raft_consensus.cc:2932] T 515491e619e145a292c5e5f654cc63ee P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.257203 4773 consensus_queue.cc:237] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:52.260890 4594 catalog_manager.cc:5484] T 515491e619e145a292c5e5f654cc63ee P 09ff1ee782864ac99d05a7f011aeef44 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "09ff1ee782864ac99d05a7f011aeef44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:52.276051 4852 raft_consensus.cc:1261] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.278029 5704 consensus_queue.cc:1035] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:39:52.281713 4738 consensus_peers.cc:485] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6e03d7dc5734e2e9e52437b5a1bc124. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:52.337563 4699 raft_consensus.cc:1261] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.349458 5704 consensus_queue.cc:1035] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:52.361387 5662 raft_consensus.cc:2932] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.362473 4847 raft_consensus.cc:2932] T e6e03d7dc5734e2e9e52437b5a1bc124 P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.394466 4696 raft_consensus.cc:2932] T e6e03d7dc5734e2e9e52437b5a1bc124 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.421646 4590 catalog_manager.cc:5484] T e6e03d7dc5734e2e9e52437b5a1bc124 P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:52.424508 4773 consensus_queue.cc:237] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
W20240502 21:39:52.435139 4738 consensus_peers.cc:485] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91e028b61d3e4a92b66697049b88e05a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:52.435163 4699 raft_consensus.cc:1261] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.435389 4847 raft_consensus.cc:1261] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Refusing update from remote peer ddea410051614f23b93c7342ec7cc545: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.436898 5704 consensus_queue.cc:1035] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 21:39:52.438992 5715 consensus_queue.cc:1035] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:39:52.460714 5692 raft_consensus.cc:2932] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.467372 4849 raft_consensus.cc:2932] T 91e028b61d3e4a92b66697049b88e05a P df8e5a877b6640d692917083c9fb5282 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.475590 4847 consensus_queue.cc:237] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } }
I20240502 21:39:52.480892 4594 catalog_manager.cc:5484] T 91e028b61d3e4a92b66697049b88e05a P ddea410051614f23b93c7342ec7cc545 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "ddea410051614f23b93c7342ec7cc545" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:52.466969 4699 raft_consensus.cc:2932] T 91e028b61d3e4a92b66697049b88e05a P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } attrs { replace: true } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.492910 4773 raft_consensus.cc:1261] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.495018 4699 raft_consensus.cc:1261] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Refusing update from remote peer df8e5a877b6640d692917083c9fb5282: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:39:52.495715 5711 consensus_queue.cc:1035] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240502 21:39:52.496757 5709 consensus_queue.cc:1035] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 21:39:52.498947 4816 consensus_peers.cc:485] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 -> Peer 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196:35981): Couldn't send request to peer 6a9bb5562dce4197b3ca1f398c12954c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 467ec4dc8d27420e9ea3fa4584153e65. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:39:52.509423 5613 raft_consensus.cc:2932] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.512646 4695 raft_consensus.cc:2932] T 467ec4dc8d27420e9ea3fa4584153e65 P 09ff1ee782864ac99d05a7f011aeef44 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.516690 4773 raft_consensus.cc:2932] T 467ec4dc8d27420e9ea3fa4584153e65 P ddea410051614f23b93c7342ec7cc545 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } } }
I20240502 21:39:52.531561 4594 catalog_manager.cc:5484] T 467ec4dc8d27420e9ea3fa4584153e65 P df8e5a877b6640d692917083c9fb5282 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a9bb5562dce4197b3ca1f398c12954c (127.3.113.196) added. New cstate: current_term: 1 leader_uuid: "df8e5a877b6640d692917083c9fb5282" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09ff1ee782864ac99d05a7f011aeef44" member_type: VOTER last_known_addr { host: "127.3.113.193" port: 44335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddea410051614f23b93c7342ec7cc545" member_type: VOTER last_known_addr { host: "127.3.113.194" port: 38679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8e5a877b6640d692917083c9fb5282" member_type: VOTER last_known_addr { host: "127.3.113.195" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9bb5562dce4197b3ca1f398c12954c" member_type: NON_VOTER last_known_addr { host: "127.3.113.196" port: 35981 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:39:52.701227 5724 ts_tablet_manager.cc:902] T e6e03d7dc5734e2e9e52437b5a1bc124 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
F20240502 21:39:52.702339 5724 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-047156e0297249998c3f2c5362c70432.queue_time_us, tablet-download-pool-047156e0297249998c3f2c5362c70432.queue_time_us) (tablet-download-pool-047156e0297249998c3f2c5362c70432.run_cpu_time_us, tablet-download-pool-047156e0297249998c3f2c5362c70432.run_cpu_time_us) (tablet-download-pool-047156e0297249998c3f2c5362c70432.run_wall_time_us, tablet-download-pool-047156e0297249998c3f2c5362c70432.run_wall_time_us) (tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.queue_time_us, tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.queue_time_us) (tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.run_cpu_time_us, tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.run_cpu_time_us) (tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.run_wall_time_us, tablet-download-pool-1c65cfc531344398b70ce5862b2211fd.run_wall_time_us) (tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.queue_time_us, tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.queue_time_us) (tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.run_cpu_time_us, tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.run_cpu_time_us) (tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.run_wall_time_us, tablet-download-pool-1cd1bcf32e7444ed80e0f7ae05786dde.run_wall_time_us) (tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.queue_time_us, tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.queue_time_us) (tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.run_cpu_time_us, tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.run_cpu_time_us) (tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.run_wall_time_us, tablet-download-pool-238a1f48ddb14387bc2aa4419ff8879e.run_wall_time_us) (tablet-download-pool-302a987ffb0842c0bfae54675c665763.queue_time_us, tablet-download-pool-302a987ffb0842c0bfae54675c665763.queue_time_us) (tablet-download-pool-302a987ffb0842c0bfae54675c665763.run_cpu_time_us, tablet-download-pool-302a987ffb0842c0bfae54675c665763.run_cpu_time_us) (tablet-download-pool-302a987ffb0842c0bfae54675c665763.run_wall_time_us, tablet-download-pool-302a987ffb0842c0bfae54675c665763.run_wall_time_us) (tablet-download-pool-80c35cadfff54236bbf05e32f945728f.queue_time_us, tablet-download-pool-80c35cadfff54236bbf05e32f945728f.queue_time_us) (tablet-download-pool-80c35cadfff54236bbf05e32f945728f.run_cpu_time_us, tablet-download-pool-80c35cadfff54236bbf05e32f945728f.run_cpu_time_us) (tablet-download-pool-80c35cadfff54236bbf05e32f945728f.run_wall_time_us, tablet-download-pool-80c35cadfff54236bbf05e32f945728f.run_wall_time_us) (tablet-download-pool-856dc0c130c04f4894884164ae24a053.queue_time_us, tablet-download-pool-856dc0c130c04f4894884164ae24a053.queue_time_us) (tablet-download-pool-856dc0c130c04f4894884164ae24a053.run_cpu_time_us, tablet-download-pool-856dc0c130c04f4894884164ae24a053.run_cpu_time_us) (tablet-download-pool-856dc0c130c04f4894884164ae24a053.run_wall_time_us, tablet-download-pool-856dc0c130c04f4894884164ae24a053.run_wall_time_us) (tablet-download-pool-96d93565463749aab6550e43871f7730.queue_time_us, tablet-download-pool-96d93565463749aab6550e43871f7730.queue_time_us) (tablet-download-pool-96d93565463749aab6550e43871f7730.run_cpu_time_us, tablet-download-pool-96d93565463749aab6550e43871f7730.run_cpu_time_us) (tablet-download-pool-96d93565463749aab6550e43871f7730.run_wall_time_us, tablet-download-pool-96d93565463749aab6550e43871f7730.run_wall_time_us) (tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.queue_time_us, tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.queue_time_us) (tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.run_cpu_time_us, tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.run_cpu_time_us) (tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.run_wall_time_us, tablet-download-pool-a1a911c9636f426e9d08bee71f60eaab.run_wall_time_us) (tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.queue_time_us, tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.queue_time_us) (tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.run_cpu_time_us, tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.run_cpu_time_us) (tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.run_wall_time_us, tablet-download-pool-df2394febdff4bb8bd495e4b8086e412.run_wall_time_us) (tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.queue_time_us, tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.queue_time_us) (tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.run_cpu_time_us, tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.run_cpu_time_us) (tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.run_wall_time_us, tablet-download-pool-e6171c5f680040b9a3efc1f98c456ea9.run_wall_time_us) (tablet-download-pool-e6e03d7dc5734e2e9e52437b5a1bc124.queue_time_us, tablet-download-pool-e6e03d7dc5734e2e9e52437b5a1bc124.queue_time_us) (tablet-download-pool-f14af5e6ab984917b771589cdb45a779.queue_time_us, tablet-download-pool-f14af5e6ab984917b771589cdb45a779.queue_time_us) (tablet-download-pool-f14af5e6ab984917b771589cdb45a779.run_cpu_time_us, tablet-download-pool-f14af5e6ab984917b771589cdb45a779.run_cpu_time_us) (tablet-download-pool-f14af5e6ab984917b771589cdb45a779.run_wall_time_us, tablet-download-pool-f14af5e6ab984917b771589cdb45a779.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714685992 (unix time) try "date -d @1714685992" if you are using GNU date ***
I20240502 21:39:52.727617 5725 ts_tablet_manager.cc:902] T 515491e619e145a292c5e5f654cc63ee P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer 09ff1ee782864ac99d05a7f011aeef44 (127.3.113.193:44335)
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800000dc7) received by PID 3527 (TID 0x7f119d48f700) from PID 3527; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f1257325980 (unknown) at ??:0
@ 0x7f1250899fb7 gsignal at ??:0
@ 0x7f125089b921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f125368fda4 google::LogMessage::SendToLog() at ??:0
@ 0x7f1253690950 google::LogMessage::Flush() at ??:0
@ 0x7f1253695a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f125401d3c7 kudu::TraceMetrics::InternName() at ??:0
I20240502 21:39:52.915911 5732 ts_tablet_manager.cc:902] T 467ec4dc8d27420e9ea3fa4584153e65 P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer df8e5a877b6640d692917083c9fb5282 (127.3.113.195:45259)
@ 0x7f1253ff394f kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f1253ff10a5 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f125be11422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f125be17c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f125bead4ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 21:39:53.043846 5742 ts_tablet_manager.cc:902] T 91e028b61d3e4a92b66697049b88e05a P 6a9bb5562dce4197b3ca1f398c12954c: Initiating tablet copy from peer ddea410051614f23b93c7342ec7cc545 (127.3.113.194:38679)
@ 0x7f125bede3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f125bec41ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f125bec417a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7f125bec410a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f125bec40d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f125bec32fe std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f125d269da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f125d269bda std::__1::function<>::operator()() at ??:0
@ 0x7f1253ff5237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1253ff8692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1253ff864a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
@ 0x7f1253ff85da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
@ 0x7f1253ff85a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7f1253ff789e std::__1::__function::__func<>::operator()() at ??:0
@ 0x7f125d269da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f125d269bda std::__1::function<>::operator()() at ??:0
@ 0x7f1253fd7937 kudu::Thread::SuperviseThread() at ??:0